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Overview 



The Bay Area School Reform Collaborative (BASRC, now called Springboard Schools) in 
San Francisco, California, is a grant-making organization that supports districts’ system-wide ef- 
forts to improve the quality and equity of student outcomes. The organization pursues various re- 
form strategies. This report discusses the “focal strategy,” which targeted six districts in the Bay 
Area (“focal districts”), beginning in the 2002-2003 school year. The strategy does not prescribe a 
particular curriculum or school structure. Instead, it promotes a vision of culture change, relying on 
three key features: coaching of district and school leaders; evidence-based decision-making at all 
levels of the system; and networking within and across schools to share experiences and lessons. 

With funding from the William and Flora Hewlett Foundation, MDRC, a nonpartisan, non- 
profit education and social policy research organization, is conducting an independent evaluation of 
BASRC’ s focal strategy. This report, the first of two, analyzes the relationship between the focal 
strategy and improvements in student achievement. It compares progress in the focal districts in the 
first two years of the strategy’s implementation to progress in a set of carefully chosen comparison 
districts in the same area over the same period. Though differences in the outcomes cannot neces- 
sarily be attributed to the BASRC focal strategy, the comparison illuminates the relationships be- 
tween student outcomes and the focal strategy. 



Key Findings 

• In the years following implementation of the focal strategy, achievement 
among third-grade students in the BASRC focal districts slightly declined, 
while achievement in the comparison districts showed no change compared 
with the baseline period. 

• On the other hand, fifth-grade students' performance in the focal districts im- 
proved over time, slightly outpacing improvements in the comparison districts 
in Year 2, but the differences were not statistically significant. 

• Among blacks and Hispanics, English Language Learners, and economically 
disadvantaged students, performance in the focal districts appeared to surpass 
the improvements in the comparison districts. The differences were most evi- 
dent in reductions in the percentage of fifth-grade students performing below 
basic levels. However, the differences were modest, generally limited to Year 
2, and not statistically significant. 

The evident lack of a substantial, pervasive association between the BASRC focal strat- 
egy and student achievement may not be surprising given that the strategy primarily targets dis- 
trict leadership and does not specify how reform activities may lead to changes in instruction or to 
instructional supports. The BASRC focal strategy has the potential to strengthen district leader- 
ship for supporting school improvement, and it may set the stage for stronger systemic improve- 
ments that are designed to change instructional practices. Thus it will be important to continue to 
look at follow-up data to ascertain whether the differences between the focal districts and the 
comparison districts — differences that were concentrated in the second year of implementation 
— persist, grow, or fade over time. 
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Preface 



A number of forces have increased the momentum for school districts to develop dis- 
trict-wide reform strategies as a means of initiating and sustaining school improvement. First, 
districts have sometimes found it easier to manage and support a single districtwide initiative 
rather than many different school reform models. Second, they have come to recognize that 
some educational problems, such as high student mobility, are better addressed above the level 
of individual schools. Finally, new standards of accountability, including requirements in the 
No Child Left Behind Act of 2002, have created an audience for new ideas about the appropri- 
ate role of districts in spearheading reform. 

The Bay Area School Reform Collaborative (BASRC), located in San Francisco, Califor- 
nia (and known now as Springboard Schools), is a grant-making organization that supports dis- 
tricts’ efforts to improve the quality and equity of student outcomes. BASRC’ s “focal strategy” is 
a district-level reform strategy being implemented in six districts throughout the San Francisco 
Bay Area. Emphasizing process rather than particular approaches, the focal strategy relies on three 
key features: coaching of district and school leaders; evidence-based decision-making throughout 
the system; and networking within and across schools to share experiences and lessons. 

This report suggests that the BASRC focal strategy is not associated with districtwide im- 
provements in average elementary student achievement. While there is the hint of a relationship 
between participation in the focal strategy and improved performance among lower-achieving, 
disadvantaged students, the differences tend to be small and are not statistically significant. More- 
over, any relationship that exists appears to be limited to the upper elementary grades. 

Given the nature of the BASRC focal reforms, the lack of a strong, pervasive relationship 
with student achievement may be understandable. In practice, the strategy primarily targets district 
leadership and does not specify how reform activities might lead to specific changes in instruction 
or specific instructional supports. Moreover, our implementation research suggests that both the 
intensity of the intervention and the consistency of focus on improving teaching and learning may 
not be sufficient. Systemic reforms such as BASRC can take a long time to take root; if they do, 
the changes in teaching and learning could be profound and more sustainable than other reforms. 
To determine if this is the case here, the next report from this evaluation will present an additional 
year of analysis and explore the relationship between schools’ implementation of particular 
BASRC reform concepts and improvements in student achievement. 

Gordon L. Berlin 
President 
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Executive Summary 



This is the first of two published reports for MDRC’s evaluation of the Bay Area School 
Reform Collaborative (BASRC), a grant-making organization in San Francisco, California. 1 
BASRC is dedicated to improving student achievement in public schools and narrowing achieve- 
ment gaps between different racial, ethnic, and socioeconomic groups. BASRC does not prescribe 
particular instructional practices, curricula, or school structures as “treatment.” Rather, the organi- 
zation promotes a vision of culture change at every level of the education system, in which teach- 
ers and administrators engage in a continuous improvement process driven by collective inquiry 
into student learning outcomes, school functioning, and teacher practice. This process-oriented 
reform strategy is designed to build professional knowledge of effective practice, mutual account- 
ability, and collaboration. BASRC hypothesizes that such changes in culture will translate into 
improved teaching and learning. However, BASRC’ s reform strategy does not specify the particu- 
lar changes in teaching practice that should evolve as a result of these processes. 

This evaluation centers on a specific BASRC initiative, the “focal strategy,” which was 
implemented in five Bay Area districts (the “focal districts”) and two to four selected schools 
within those districts (the “focal schools”) during the 2002-2003 school year. (During the 2003- 
2004 school year, BASRC added a sixth focal district that is not included in MDRC’s study.) 
Building on earlier phases of BASRC reforms, which began in 1996, the focal strategy is meant 
to increase the intensity of the core BASRC reforms by creating more opportunities for district 
and school administrators to interact with BASRC staff. Thus, compared with the earlier reform 
strategies, the focal strategy serves as a stronger test of BASRC reform ideas in fewer places. 
Also, in selecting the focal districts, BASRC tended to focus on districts where there already 
was a strong working relationship developed during earlier BASRC efforts. Therefore, in many 
cases, focal reforms have essentially been implemented on top of existing BASRC reform work. 

The BASRC focal strategy has three main features: 

• coaching of superintendents, district and school leaders, and teachers 

• evidence-based decision-making at all levels of the system 

• networks and collaboration among administrators and teachers, within and 
across districts and schools 



'After this study was conducted and the report was written, BASRC changed its name to Springboard 
Schools. For ease of reference, the report uses the former name throughout. 
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All three features were also part of BASRC’s earlier reform efforts. Once the focal 
strategy began, school-level coaching by BASRC staff was redirected toward focal districts. In 
addition, “executive coaches” (former superintendents) were hired to advise and support the 
leaders in the focal districts. Along with regular meetings on comprehensive needs assessments 
and to review progress, these efforts were intended to reinforce all elements of the strategy in 
the focal districts and schools. 

The theory of action underlying the BASRC focal reform strategy posits that the imple- 
mentation of these three design features will yield changes in district, school, and classroom prac- 
tices that will in turn improve student outcomes. Together, coaching, evidence-based decision- 
making, and networks and collaboration are thought to increase districts’ and schools’ engage- 
ment in a continuous improvement process, focused on improving the level and equity of student 
achievement levels. Because the strategy is process-oriented, it can result in different outcomes 
within each district, school, and classroom. The outcomes may or may not be policies or practices 
that are directly linked with specific strategies for improved teaching and learning. They may be 
incremental improvements in culture that eventually foster better teaching and learning. 



The Evaluation of the BASRC Focal Strategy 

The independent evaluation of BASRC is funded by the William and Flora Hewlett 
Foundation and includes studies by both MDRC and the Stanford University Center for Re- 
search on the Context of Teaching (CRC). 2 The CRC studies focus on the process of reform as 
it relates to BASRC in general and the relationship between various BASRC reform efforts and 
changes at the district, school, and classroom levels. MDRC’s study attempts to shed light on 
the relationship between the BASRC focal strategy and improvements in student achievement 
by investigating the following: 

1 . The relationship between participation in the BASRC focal strategy and stu- 
dent achievement. 

2. The relationship between implementation of specific BASRC reform prac- 
tices and changes in student achievement. 

This report focuses on the first of these issues, that is, documenting the empirical rela- 
tionship between BASRC participation and changes in student achievement. As such, it ad- 
dresses three questions: 

• What is the BASRC focal strategy reform model/theory of action? 



2 CRC also conducted an evaluation of BASRC’ s first phase of reform work ( 1995-2001). 
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To what extent was this model implemented at the districts involved in this 
evaluation? 



• What is the association between districts’ participation in BASRC focal re- 
forms and measured changes in average student achievement? 

While MDRC’s study will reflect on the relationship between the BASRC focal strategy 
and student achievement, it cannot identify a causal relationship. Due to the nature of the BASRC 
focal strategy, and the selection process for districts’ participation in the strategy, it is not feasible 
to render a reliable, unbiased single “net impact” estimate summarizing BASRC’ s effect on stu- 
dent learning. The analysis presented in this report compares progress in the focal districts in the 
first two years of the focal reforms to progress in a set of carefully chosen comparison districts 
from throughout the San Francisco Bay Area over the same period of time. BASRC selected dis- 
tricts based on a variety of immeasurable factors, including the extent to which the superintendent 
was reform-minded. Therefore, any differences in progress in the focal districts versus that in 
other districts cannot necessarily be attributed to the BASRC focal strategy. 

The BASRC focal reforms tended to be focused on literacy instruction and concentrated 
at the elementary level. As such, all analyses are focused at the elementary school level and em- 
phasize district performance on measures of student literacy. 

MDRC’s next report, scheduled for later in 2006, will explore BASRC’ s theory of action 
by examining the relationship between changes in student achievement and schools’ implementa- 
tion of the practices the reform strategy was designed to encourage. In other words, regardless of 
why schools experience changes in reform practices, MDRC’s next report will attempt to ascertain 
whether those reform practices are correlated with differences in students’ learning. 



Key Findings 

Implementation Findings 

• By Year 2, the school-level aspect of the focal strategy faded. Thus, the 
model evolved to be almost entirely a district strategy in which focal districts 
received district-level coaching from an executive coach, some additional 
coaching, and review meetings with BASRC staff. 

In theory, coaching by BASRC staff was to be a primary feature of the school-level focal 
strategy. This “external” coaching effort was distinct from the other “internal” coaching efforts in 
place in non-focal schools (including school-level literacy coaches and coaches employed by the 
district to support reform work). However, in the first year of implementation, BASRC encoun- 
tered several complications, including resistance to the BASRC coaches from school-level staff. 
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By the end of the first year, BASRC coaches did not have a significant coaching role in most of 
the focal schools. At the district level, on the other hand, the key components of the focal strategy 
were in place by Year 2. BASRC executive coaches met with district leaders on a regular basis, 
and district staff attended networking and review meetings led by BASRC staff. 

• The extent to which the BASRC focal strategy at the district level translated 
into specific reform activities in these districts is not yet clear. 

The coaching delivered by BASRC executive coaches and other BASRC staff was not 
necessarily focused on implementation of the core BASRC concepts. Instead, in practice, 
coaching often revolved around a variety of needs of the superintendent or other district staff. 
This varied from improving the superintendent’s communication skills to advising on the im- 
plementation of a new districtwide curriculum. The extent to which coaching or meetings with 
BASRC staff translated into specific reform activities in these districts is not yet clear. Gather- 
ing evidence on activities resulting from participation in the BASRC focal strategy is a priority 
for future MDRC field work. 

• Although there was evidence of all three key features of the BASRC focal 
strategy in schools in the BASRC districts, these instances of the key features 
were likely vestiges of earlier BASRC reform phases. In addition, it was dif- 
ficult to detect meaningful differences in the types of BASRC supports and 
reform activities in place in focal schools compared with non-focal schools. 

MDRC found evidence of all three of the key features of the BASRC focal reform strat- 
egy — coaching by district or school staff, evidence-based decision-making, and networks and 
collaboration — in place at the schools in the focal districts. However, field research data indicate 
that these were typically implemented as a result of participation in earlier BASRC reform efforts 
rather than in the BASRC focal strategy itself. Moreover, it is also possible that these reform prac- 
tices were in place before any participation in BASRC reform efforts. In general, in the schools 
visited by MDRC, it was difficult to detect meaningful differences in the types of BASRC sup- 
ports and reform activities in place in focal schools compared with non-focal schools. 

Student Achievement Outcomes 

In order to explore the relationship between the BASRC focal strategy and student 
achievement, MDRC’s analysis of student achievement compares progress in the BASRC focal 
districts in the first two years of the BASRC focal reforms to progress in a set of carefully chosen 
comparison districts from throughout the San Francisco Bay Area over the same period. 

• In the years following implementation of the BASRC focal strategy, third- 
grade students in the BASRC focal districts and in similar districts 
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throughout the Bay Area showed no progress in student achievement com- 
pared with the baseline period. In fact, average proficiency rates declined 
during this timeframe. 

Neither BASRC focal districts nor their comparison counterparts experienced progress in 
third-grade proficiency rates on the California Standards Tests (CSTs). As shown in the top panel 
of Figure ES. 1, in the two years preceding the BASRC focal strategy, an average of 43 percent of 
third-grade students in the BASRC focal districts scored proficient or above on the language arts 
portion of the CST. By the end of the second year of implementation, this dropped to 37 percent. 
Over the same period, average proficiency rates in similar districts throughout the Bay Area and 
across the state dropped slightly as well. Also, as shown in the lower panel of Figure ES.l, the 
percentage of third-grade students scoring below basic remained relatively constant in the com- 
parison districts and increased slightly in the focal districts. 

• On the other hand, fifth-grade students’ performance in the focal districts im- 
proved over time, slightly outpacing improvements in the comparison dis- 
tricts, but the differences were not statistically significant. 

As shown in the top panel of Figure ES.2, in the years immediately preceding implemen- 
tation of the focal strategy, proficiency rates on the CST language arts test averaged 38 percent 
among fifth-graders in BASRC focal districts. This rate increased to 5 1 percent by the end of the 
second year of focal strategy implementation. At the same time, proficiency levels in the compari- 
son districts increased from 39 percent to 50 percent. Although the increases were slightly larger 
in the BASRC focal districts, these differences were modest in size and not statistically signifi- 
cant. As shown in the lower panel of Figure ES.2, there was also a reduction in the percentage of 
fifth-grade students performing below basic. Again, although these reductions were slightly larger 
in the BASRC focal districts, the differences were not statistically significant. Rather than reflect- 
ing systematic differences between progress in the focal districts and progress in the non-focal 
districts, they may reflect chance or “random” fluctuations in student outcomes. 

• Among blacks and Hispanics, English Language Learners (ELL), and eco- 
nomically disadvantaged students, reductions in the percent of fifth-grade 
students performing below basic levels in the BASRC focal districts in Year 
2 outpaced the reductions in low-performing fifth-graders at the comparison 
districts. However, these differences were not statistically significant. 

Across the BASRC focal districts, for each of these subgroups, there was a reduction in 
the percentage of fifth-grade students scoring below basic that outpaced the average reduction in 
the comparison districts. These differences were not statistically significant and were generally 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 

Figure ES.l 

Third-Grade Student Performance on the California Standards Test, Language Arts 



Percentage Scoring Proficient or Advanced 




Baseline Year 1 Year 2 Baseline Year 1 Year 2 

F ocal Districts Comparison Districts 




Baseline Year 1 Year 2 Baseline Year 1 Year 2 



Focal Districts Comparison Districts 

SOURCE: MDRC calculations from district and school records from the California Department of 
Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 
2000-2001 and 2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003- 
2004) is the second follow-up school year. The estimates in the table represent averages across all 
districts, regardless of size. 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 

Figure ES.2 

Fifth-Grade Student Performance on the California Standards Test, Language Arts 



Percentage Scoring Proficient or Advanced 




Baseline Year 1 Year 2 Baseline Year 1 Year 2 



Focal Districts Comparison Districts 



Percentage Scoring Below Basic 




Focal Districts Comparison Districts 



SOURCE: MDRC calculations from district and school records from the California Department of 
Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 
2000-2001 and 2001-2002. Year 1 (2003-2003) is the first follow-up school year and Year 2 (2003- 
2004) is the second follow-up school year. The estimates in the table represent averages across all 
districts, regardless of size. 
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limited to Year 2 of the intervention. However, the fact that the pattern repeats itself across sev- 
eral subgroups suggests the possibility that the BASRC focal districts may have made progress 
in improving the performance of their lowest-performing fifth-graders. The improvements were 
largest among ELL students (not shown in the figures). Lor example, during the baseline period, 
55 percent of fifth-grade ELL students scored below basic in the focal districts. By the second 
year of follow-up, only 41 percent scored below basic, a reduction of 14 percentage points. The 
comparison districts, in contrast, experienced less improvement, reducing the proportion of stu- 
dents scoring below basic by 5 percentage points. 



Interpreting the Findings 

The evidence presented in this report suggests that the BASRC focal strategy is not as- 
sociated with improvements in achievement among third-graders. However, with respect to 
fifth-graders, the progress of the BASRC focal districts tended to outpace that of the compari- 
son groups, particularly among lower-performing disadvantaged, minority, and ELL students. 

It is important to note that these differences tended to be small, and were not statisti- 
cally significant. As such, it may be that there are no true differences between the progress in 
the focal and non-focal districts. Moreover, the design of the study does not prove a causal rela- 
tionship between participation in the BASRC focal strategy and improved student outcomes for 
lower-performing students. However, it is possible that focal reforms had a modest effect on 
student performance by lower-achieving fifth-graders. Since this relationship existed primarily 
in the second year of implementation, it will be important to examine follow-up data to ascer- 
tain whether these differences persist, grow, or fade over time. 

What explains the evident lack of a substantial, pervasive association between the 
BASRC focal strategy and student achievement? The implementation and outcome findings 
suggest several possible interpretations: 

Hypothesis 1 : The BASRC focal reforms are not intense enough to affect stu- 
dents’ academic performance. 

While the core components of BASRC focal reforms may be potential drivers of im- 
proved teaching and learning, the implementation of the focal reform strategy may simply not 
have been intense enough to change student achievement in a pervasive manner. Lor example, 
there may not have been a sufficient number of interactions between BASRC staff and district and 
school leaders, or these interactions may not have been sufficiently focused on implementation of 
the BASRC reform strategies. The fact that the school-level strategy faded away may have limited 
the intensity of the reform activities. A lack of intensity could also be the result of a lack of focus, 
which could lead to not spending very much time on any particular effort. To the extent that this is 
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true, it might suggest that, for the focal reforms to have an effect, BASRC must increase the inten- 
sity of its district-reform efforts and imbue those efforts with a consistent focus. 

Hypothesis 2: BASRC focal reforms are not reaching the classroom. 

It is also possible that the BASRC focal strategy, regardless of intensity (that is, regard- 
less of frequent and focused coaching sessions with the superintendent), would not lead to im- 
provements in teaching and learning because the intervention is not close enough to the class- 
room. This might suggest that, in order to affect teaching and learning, the focal strategy must 
intervene in ways that ensure reforms reach the classroom level. This could include sustaining 
school-level coaching efforts or providing other supports designed to increase the effectiveness 
of school- or classroom-level BASRC reforms. It is also possible that, by their nature, district 
reforms can be effective but simply require more time to take root at the school level. 

Hypothesis 3: Core BASRC reforms are not sufficiently powerful drivers of im- 
proved teaching and learning. 

On the other hand, even if the focal strategy did increase the intensity of reform efforts at 
the school and classroom levels, it is possible that the reforms themselves are not strongly related 
to improved student achievement. In particular, it is possible that the BASRC reform activities 
supported by the focal strategy, even when effectively implemented, do not result in measurable 
improvements in teaching and learning. This would suggest that the BASRC focal districts and 
schools would make more progress by implementing reforms focused more directly on refining 
classroom practice or by implementing particular pedagogical approaches. MDRC’s next report 
will explore this hypothesis by examining the relationship between school-level implementation 
of particular BASRC reforms and changes in student achievement. 
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Chapter 1 

Introduction 



This is the first of two reports for MDRC’s evaluation of the Bay Area School Reform 
Collaborative (BASRC), a grant-making and support organization in San Francisco, California. 1 
Although BASRC has launched several reform efforts since its inception, this evaluation is fo- 
cused on a particular initiative, the “focal strategy,” which was implemented in Bay Area districts 
starting in the 2002-2003 school year. Guided theoretically by the use of data, the focal strategy is 
a reform effort aimed at building capacity across entire districts and at all levels of the system for a 
systematic and continuous education improvement process. The focal strategy also concentrates 
BASRC staff resources in a small number of districts, and in schools within those districts, in or- 
der to provide intensive and responsive coaching support to teachers. 

With an increasing momentum for school districts to develop systemwide reform 
strategies as a means of initiating and sustaining school improvement, the evaluation of BASRC 
provides a timely opportunity to expand knowledge in the field. This report provides a first look 
at how the BASRC focal strategy has unfolded and analyzes the relationship between participa- 
tion in the focal strategy and trends in student achievement. The key question driving this report 
is whether the progress in districts that participated in the focal reforms has outpaced that of 
similar districts from the Bay Area that did not participate in these reforms. Because some of the 
participating districts and some of the non-participating comparison districts also took part in 
earlier BASRC reforms, this report assesses the relationship between the BASRC focal strategy 
and student achievement, apart from any changes associated with previous BASRC reforms (or 
other ongoing reforms unrelated to BASRC). In other words, the districts under evaluation are 
unique in their participation in the BASRC focal strategy, but not necessarily in their participa- 
tion in BASRC or in their participation in reforms more broadly. 2 



Overview of BASRC Reforms 

BASRC is dedicated to improving student achievement in public schools and narrowing 
achievement gaps between different racial, ethnic, and socioeconomic groups. BASRC pro- 
motes a vision of culture change in which teachers and administrators engage in a collaborative 



'After this study was conducted and the report was written, BASRC changed its name to Springboard 
Schools. For ease of reference, the report uses the former name throughout. 

“BASRC reforms were implemented in numerous districts throughout the Bay Area. If the districts that 
did not participate in any prior BASRC reforms were eliminated from the evaluation, there would not be a suf- 
ficient sample of districts to estimate a relationship between the focal strategy and student achievement. 
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and ongoing reform process at every level of the education system. BASRC’s reforms do not 
prescribe predetermined instructional practices, curricula, or school structures as a “treatment.” 
Rather, they are designed as processes of continuous improvement, driven by collective inquiry 
into student learning outcomes, school functioning, and teaching practices. According to 
BASRC, its process-oriented reform strategy builds professional knowledge of effective prac- 
tice, mutual accountability, and collaboration, and continuous improvement of the quality and 
equity of student outcomes. Importantly, while BASRC hypothesizes that these changes in cul- 
ture will translate into improved student learning, the organization does not specify the specific 
mechanisms or pathways through which such changes translate into changes in teaching. Nor 
does it specify the specific changes in teaching that should evolve as a result of these processes. 

BASRC reforms are distinct from prescriptive classroom-level reforms such as “Suc- 
cess for All,” which aim to implement particular changes in instructional practice in order to 
improve student achievement and reduce racial and economic achievement gaps. 3 Though 
BASRC reforms are implemented at both the school and the district level, BASRC is conceptu- 
ally closer in approach to the initial idea of the “Accelerated Schools” model, in that it is not 
built around a single definition of effective teaching practice, but instead draws on coaches to 
facilitate a process for school improvement that is intended to improve teaching. 4 

BASRC reforms have been implemented in Bay Area schools and districts in several 
phases over the past nine years (see Figure 1.1). Initially incorporated in early 1995, BASRC 
received $50 million in matched grants from the William and Flora Hewlett Foundation and the 
Annenberg Foundation. During Phase I (1995-2001), BASRC awarded grants to 87 schools 
over three to five years. At this stage, BASRC reforms focused on school-level change and cen- 
tered on inquiry as the engine for improving school practice and student achievement. 5 In 2001, 
the organization received an additional $40 million ($25 million and $15 million from the two 
foundations, respectively) to embark on its second phase of reform work in the Bay Area. In 
Phase II (2001-2006), BASRC began to focus more on the district as an agent for change and 
for scaling up reform. BASRC expanded its focus to a coordinated school- and district-level 
reform model in response to feedback from the Phase I evaluation that schools could not sustain 
improvement without active district support. 6 As part of Phase II, BASRC awarded grants to 18 
districts to support 23 clusters of schools within those districts (a total of 91 schools) in working 
together on reform efforts. These districts, selected from a pool of Bay Area grant applicants, 



3 Slavin, Madden, Dolan, and Wasik (1993). 

4 Bloom (2002). 

5 Center for Research on the Context of Teaching (2002). 

6 Masten and Rendell (2002). 
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Figure 1.1 

Timeline of BASRC Reform Efforts 




received annual renewable, multiyear improvement grants, which ranged from $125,000 to 
$500,000 (depending on the number of BASRC schools in the BASRC district grant). 7 

While Phase II was underway, during the 2002-2003 school year, BASRC also em- 
barked on a “focal strategy.” This strategy directed BASRC staff resources to six “focal dis- 
tricts” and 19 “focal schools” within those districts. 8 While the focal districts and schools did 
not receive any additional BASRC funding with the introduction of the focal strategy, they did 
receive increased staff coaching time. (BASRC decreased individualized support for its broader 
pool of grantees in order to reallocate more staff time to the focal sites.) Table 1.1 summarizes 
the key dimensions of BASRC Phase I, Phase II, and the focal strategy. 

While the initiatives were in operation, BASRC maintained a consistent vision of rais- 
ing overall achievement and closing achievement gaps; however, the strategy for attaining this 
goal changed over time. BASRC’ s focus, or unit of change, shifted first from working with in- 
dividual schools to groups of schools working together (referred to as Local Collaboratives), 

7 BASRC also awarded giants of $2.1 million over three years to another nine school districts to support 
them in research and development efforts aimed at discovering how to change their own policies and practices 
to better support school improvement. 

8 Phase II schools may or may not have been funded in Phase I. Likewise, focal schools may or may not 
have been funded at the start of Phase II. 
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Table 1.1 

Comparison of BASRC Reform Efforts 





Phase I 


Phase II 






Ongoing Phase II 


Focal Strategy 


Years 


(1996-1997 to 2000-2001) 


(2001-2002 to 2004-2005) 


(2002-2003 to 2004-2005) 


Vision 


Raise achievement and 
close achievement gaps 


Raise achievement and close 
achievement gaps 


Raise achievement and close 
achievement gaps 


Unit of Change 


Schools 


Local collaboratives (clusters 
of schools within district and 
district office) 


Districts 


Breadth 


87 schools 


23 local collaboratives (91 
schools in 18 districts) 


6 focal districts and 19 focal 
schools within them 


Key Dimensions 
of Reform 


- Evidence -based deci- 
sion-making 

- Distributed Leadership 

- Professional community 


Coaching 

Evidence -based deci- 
sion-making 
Networks and collabora- 
tion 


Coaching 

Evidence -based deci- 
sion-making 
Networks and collabora- 
tion 


Delivery System 








BASRC Staff 
Coaching 


Direct assistance to leader- 
ship schools not achieving 
review-of-progress (ROP) 
goals 


Available to all grantee 
schools (2001-2002 only) 


Executive coaching and 
BASRC staff coaching of 
district leaders. BASRC staff 
coaching of all focal schools 
within focal districts 3 


Meetings 


Summer institutes 
Network meetings by topic 
(for example, data, assess- 
ment, literacy) 


Summer institutes 
Network meetings by roles 
(for example district admin- 
istrators, principals and 
coaches) 


Summer institutes 
Focal district network meet- 
ings 

Network meetings by roles 


Accountability 


Poitfolio review 
Review of Progress (ROP) 
(annual review meetings) 


Review of Progress (ROP) 


Quarterly reviews 
Review of Progress (ROP) 



NOTES: “School-level coaching by BASRC staff was discontinued early in the process of implementing the 
BASRC focal strategy. As such, most of the coaching in the focal strategy occurred at the district level. 
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and then to districts (and schools within those districts). The reforms developed over these 
phases, but remained centered on pursuing evidence-based decision-making (BASRC’s Cycle 
of Inquiry) and building collaboration. During Phase I, BASRC encouraged schools to employ 
outside support providers, but did not initially engage in direct technical assistance. After ac- 
knowledging that some schools needed assistance to achieve their goals, BASRC started offer- 
ing school-level coaching toward the end of Phase I. 9 The coaching component became an inte- 
gral part of BASRC’s delivery strategy in Year 1 of Phase II and, in turn, in the focal strategy. 
In fact, BASRC developed the focal strategy to offer more support to districts and schools than 
had been offered in earlier phases of reform. In this way, BASRC concentrated and expanded 
many of the ideas developed for Phase II. The focal strategy was designed to increase the extent 
to which BASRC reform ideas would be carried out, by increasing contact with BASRC staff, 
particularly at the district level. 

In selecting the focal districts, BASRC tended to look to districts where there was al- 
ready a strong working relationship from Phase I and/or Phase II efforts. (As a result, four of the 
six focal districts were part of earlier BASRC efforts.) Districts with achievement gaps and with 
a reform-minded superintendent were also high priorities. The focal schools were then selected 
through negotiations between BASRC and district administrators. 

Like the original Phase II reform model, the BASRC focal strategy emphasizes system- 
level reform and is built on three primary design features: 

(1) coaching 

(2) evidence-based decision-making 

(3) professional networks and collaboration 

However, by 2002-2003 (Year 2 of Phase II and the first year of the focal strategy), coaching by 
BASRC staff or individuals selected and trained by BASRC was dedicated primarily to focal 
districts and focal schools. Coaching by district or school staff, data-based decision-making, and 
networks are each elements of BASRC’s original Phase II reform model. However, in focal dis- 
tricts and schools these effects were supposed to be reinforced by interactions with BASRC 
staff. All of these features will be discussed in more detail in Chapter 2. 



9 Center for Research on the Context of Teaching (2002). 



5 




The Evaluation of the BASRC Focal Strategy 

The independent evaluation of BASRC is a collaborative effort of MDRC and the Stan- 
ford University Center for Research on the Context of Teaching (CRC). 10 MDRC’s study will 
shed light on the relationship between the BASRC focal strategy and improvements in student 
achievement. This report will show the degree of academic progress in BASRC focal districts, 
and whether or not it outpaces progress in similar districts not participating in the BASRC focal 
strategy. In future analyses, MDRC’s study will also demonstrate whether or not school reform 
practices that are consistent with the BASRC focal theory of action are actually correlated with 
improvements in student performance. BASRC focal reforms are focused on literacy instruction 
and are concentrated at the elementary level. As such, all analyses are focused at the elementary 
school level and emphasize district performance on measures of student literacy. 

MDRC’s study does not isolate the effect of BASRC focal strategy on student achieve- 
ment. Due to the nature of the BASRC focal strategy, and the nature of districts' participation in 
the intervention, it is not feasible to render a single reliable and unbiased “net impact” estimate 
summarizing BASRC’ s effect on student learning. Assessing the net impact of any educational 
reform requires a reliable estimate of outcome levels that would have occurred in the absence of 
the reform. As will be explained in more detail in Chapter 4, because districts are not selected at 
random, this is simply not possible in the case of the BASRC focal strategy. Nevertheless, 
MDRC’s study will be able to reflect on the relationship between the BASRC focal strategy and 
improvements in student achievement. 

This study will inform and is informed by the CRC studies. Its studies focus in more de- 
tail on the process of reform and the relationship between BASRC reform efforts and changes at 
the district, school, and classroom levels. CRC’s studies will generate detailed information 
about the implementation of (1) particular design features, (2) the school and district contexts in 
which these reforms take place, and (3) how both the reform efforts and the reform contexts 
evolve over time. Moreover, information provided by the CRC studies will facilitate the inter- 
pretation of the findings of the MDRC study (for example, descriptions of the factors driving 
the changes in particular reform practices, or the dynamics limiting or enhancing the connec- 
tions between particular reform practices and improved student achievement). Together, the 
MDRC and CRC studies will contribute to the knowledge base for local system reform by ex- 
ploring both the process of system reform and the relationship between particular reform prac- 
tices and changes in student achievement. 



I0 CRC also conducted an evaluation of BASRC’ s first phase of reform work (1995-2001). (See Center for 
Research on the Context of Teaching, 2002.) 
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Overview of This Report 

This is the first of two published reports in MDRC’s evaluation of BASRC. This report 
sets out to answer the following question: “What is the relationship between participation in the 
BASRC focal strategy and student achievement?” As the evaluation proceeded, the researchers 
refined their research questions according to lessons learned in the field. Primarily, through inter- 
views with principals, coaches, and teachers in the focal schools, as well as with members of 
BASRC staff, the researchers learned that the focal strategy at the school level was not imple- 
mented as planned." Because implementation of the school-level part of the focal strategy was 
weak, it no longer made sense to compare achievement trends between “focal” and “non-focal” 
schools. This report therefore focuses on district-level trends, and evaluates how changes in stu- 
dent performance in BASRC focal districts compare with changes in student performance in simi- 
lar districts over the same period. In particular, this report focuses on the following two questions: 

• How does student performance in BASRC focal districts compare with student 
performance in those districts prior to the implementation of focal reforms? 

• How do these changes in student performance in focal districts compare with 
the changes in student performance in other similar districts over the same 
period? 

The findings presented in this report do not indicate a systematic relationship between 
the BASRC focal strategy and districts’ progress in average reading achievement among third- 
graders. In fact, achievement among third-graders declined by slightly more in the BASRC fo- 
cal districts than in similar districts throughout the San Francisco Bay Area. On the other hand, 
the findings do suggest the possibility that there is an association between districts’ participation 
in the BASRC focal strategy and improved achievement among fifth-graders, particularly 
among key subgroups. These key subgroups include economically disadvantaged students, 
black and Hispanic students, and English Language Learners. However, as these differences 
were small and not statistically significant, MDRC cannot rule out the possibility that these as- 
sociations were driven by chance. 

The remainder of this report is organized as follows. Chapter 2 explains the theory of ac- 
tion behind the BASRC focal strategy. Chapter 3 presents characteristics of BASRC focal districts 
before they started participating in the focal strategy and provides a summary of implementation 
issues during the first two years. Chapter 4 describes the analytic approach and presents prelimi- 
nary findings on the relationship between the BASRC focal district strategy and trends in student 
achievement, for all students and for subgroups of students. Linally, Chapter 5 discusses hypothe- 
ses for explaining the findings and raises questions for further analysis and reporting. 

"This is discussed in greater detail in Chapter 3. 
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A subsequent report will examine the connections between specific school- and class- 
room-level reform practices and school-level trends in student achievement. Those analyses will 
draw on three years of data from a survey of teachers at elementary schools. The report will 
shed light on how variation in changes in school reform and organizational practices related to 
BASRC correlate with student achievement outcomes. It will assess the empirical relationships 
underlying BASRC’ s theory of action and provide valuable insights for district leaders and re- 
form partners about the types of promising practices upon which they might want to focus. This 
second paper will also draw on field research conducted during the second and third years of 
focal strategy implementation to add context and interpretation to the findings. Lastly, the paper 
will update the findings presented in this report with one additional year of achievement data. 




Chapter 2 

The BASRC Focal Strategy 



Theory of Action 

Since the inception of its focal strategy in fall 2002, the Bay Area School Reform Col- 
laborative (BASRC) has continuously refined the theory of action underlying the focal strategy 
in order to be responsive to changing local needs and priorities. However, three key features 
have remained central: 

• Coaching 

• Evidence-based decision-making 

• Professional networks and collaboration 

This section describes these core features of the focal strategy, how they were originally 
intended to be put into practice in focal districts and schools, and the underlying theory linking 
them to improved achievement and accelerated learning among the lowest-performing students. 
This section also includes a description of BASRC’ s “blueprint” for the focal strategy, and what 
might be observed if the strategy were to be implemented as planned. 1 Chapter 3 offers observa- 
tions based on actual implementation of the focal strategy. 

As mentioned earlier, it is important to note that the focal strategy is very similar to the 
Phase II strategy in terms of the key components. What sets the focal theory apart from the 
Phase II theory is that BASRC provides districts and schools with more intensive support for 
reform, primarily in the form of coaching by BASRC staff and experienced coaches hired by 
BASRC to support district reform. To support the work of the coaches, BASRC also conducts 
comprehensive needs assessments in the focal districts and quarterly meetings to review pro- 
gress. Coaching (from within the school system), evidence-based decision-making, and net- 
works, established as the key components for reform in Phase II, are also the key components of 
the focal strategy. 



'The ongoing changes in BASRC’s model and implementation make this challenging, but this evaluation 
focuses on the basic features that appear to remain stable in BASRC’s design over time. 
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Coaching 

BASRC has based its underlying theory of coaching on the idea that educators “need 
help finding, prioritizing, and implementing the many good ideas available [in the field ].” 2 
Coaching was intended to be the primary delivery system for BASRC focal reforms. BASRC’ s 
original focal strategy included coaching at both the district and school levels. At the outset of 
their participation in the focal strategy, both districts and schools set measurable goals for rais- 
ing student achievement and narrowing the performance gaps between different groups of stu- 
dents. BASRC coaches were supposed to work with leaders at both levels to keep them focused 
on these goals and to help them develop and implement strategies for achieving the goals. 

The BASRC theory is that, by providing “practical, useful, and effective support,” their 
coaching will lead to better teaching and improved student achievement . 3 According to 
BASRC, by “diagnosing problems, identifying organizational areas of need, setting goals and 
agreeing on a focus, establishing an instructional plan based on research and best practices in 
high-performing schools and districts, and putting in place a rigorous and thoughtful continuous 
improvement process,” coaching can lead to improved teaching and learning . 4 The coaching 
model does not, however, call for any specific activities designed to ensure particular changes in 
teaching practice, nor does it stipulate how these changes are to occur. 

At the district level, BASRC h ir ed executive coaches with district leadership experience 
to work with superintendents and/or other central office administrators. Their coaching was in- 
tended to address the district’s capacity with respect to five core elements defined by BASRC : 5 

• Leadership: The superintendent and other district leaders should provide and 
articulate a vision for a districtwide focus. They should support that vision by 
allocating resources, providing schools with flexibility, and making sure all 
school staff members have the instructional materials, technology, and tools 
they need to achieve agreed-upon organizational goals. 

• Culture: Teachers and administrators throughout the district should believe 
that every student in the district, regardless of gender, race, primary lan- 
guage, or socioeconomic status, is capable of meeting district standards, 
when provided with effective instruction. Teachers and administrators should 
strive for and reward excellence, make the best use of time during the school 



2 BASRC Web site (2003). 

3 BASRC Web site (2005). 

4 BASRC Web site (2003). 

5 These terms and their definitions summarize the BASRC document entitled “District Critical Ele- 
ments” (2002). 
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day, ensure safe and clean environments, and involve the community in the 
life of the school. 

• Infrastructure: The Board of Trustees and district leaders should adopt re- 
form policies that work, distribute authority and resources properly, and 
regularly evaluate progress toward goals. These leaders should also build 
strong professional learning communities, establish effective two-way com- 
munications systems, and provide well-developed professional development 
programs. 

• Educational Program: District leaders should define content standards for 
what students should know and performance standards for how students 
should demonstrate their achievement. They should also adopt or develop a 
comprehensive K-12 curriculum aligned with state and national standards 
and reinforced by evidence-based practice, define standards for effective im- 
plementation of programs and practices, and regularly examine data to de- 
termine program effectiveness. 

• Professional Development: District leaders should design professional de- 
velopment that is aligned with the district focus, and targeted to improving 
instruction and accelerating the achievement of low-performing students. 

These professional development opportunities should be regularly evaluated 
and adjusted to better meet the needs of teachers, and they should focus on 
helping teachers and administrators understand the implementation of new 
programs and strategies and use data to inform instruction. 

Executive coaches plan their activities by first leading a needs assessment of the dis- 
trict’ s capacity in each of these areas, and then helping district leaders formulate goals based on 
the results. An important aim of the executive coaching model is to help district leaders become 
reflective and outcome-oriented. Executive coaches, along with BASRC staff, also lead quar- 
terly review meetings to review and discuss district reform progress. This model assumes that 
the district’ s organizational culture and the behavior of its leaders can be leveraged in order to 
put effective reform practices into place in schools. BASRC does not, however, prescribe the 
specific pathways or mechanisms that will translate these behaviors into actual school-level 
changes in teaching and learning. 

Executive coaches were h ir ed to work with superintendents and other leaders at the dis- 
trict level. At the school level, the original design was that BASRC staff would deliver coaching 
to support schools’ efforts to achieve their reform goals. In collaboration with the executive 
coaches, BASRC school coaches were to address any number of school organizational issues 
related to district initiatives, from leadership to performance monitoring. For example, if a dis- 
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trict embarks on an initiative to implement a new assessment program, BASRC’s school- level 
coaching might focus on helping a school use that assessment data in an analysis of student 
achievement. BASRC school-level coaches work with principals, school- and district-based 
coaches, grade-level teams, and/or other leadership teams. (As Chapter 3 will discuss, because 
of implementation challenges, school-level coaching faded from the focal model in many 
schools relatively early in the implementation process.) 

BASRC coaches provide focal districts and schools with specific tools and supports de- 
veloped by BASRC. An example of one such tool is a database developed by Just for the Kids - 
California (JFTK-CA). The database provides districts and schools with state standards assess- 
ment results by grade, and includes average results for all students as well as for students who 
have been in the school for at least three years. For any school in the system, JFTK-CA also 
provides the names of the top 10 performing schools in California with comparable student 
demographics. The vision for this system is that a district’ s leaders and teachers will communi- 
cate with these highest-performing schools in order to learn about practices that drive their suc- 
cess. Another example of a tool coaches may use is a worksheet that walks leaders and teachers 
through a Cycle of Inquiry process. The worksheet includes questions for educators to answer, 
which allow them to identify a problem, diagnose it, and develop a practice to address it. 

Coaching by the BASRC staff is a delivery system for the BASRC focal reforms. At 
the same time, coaching by staff within the school system is also a key feature of the BASRC 
focal reforms and of the earlier phases of BASRC. 

Three types of coaching roles were originally specified as part of the broader Phase II 
strategy. These remain as part of the focal strategy as well. First, Local Collaborative coaches 
grew out of the focus on groups of schools working together on reform issues (Local Collabora- 
tives) in Phase II. These coaches, based either in the district office or at a school site, bridge 
BASRC schools with one another and the district office, promoting both bottom-up and top- 
down reform work. They may be district administrators, principals, school-based coaches, or 
individuals hired by the district specifically for the role. Second, reform coordinators are based 
on-site at BASRC schools and are often teachers or other school staff who are released part time 
to help advance BASRC reforms. Third, literacy coaches are typically part of a district strategy 
not necessarily related to BASRC. Literacy coaches help with the adoption of new curriculum 
or provide intervention work for struggling students. Table 2.1 lists the various coaching roles, 
their organizational affiliations, and their relationship to the different phases of BASRC. 



Evidence-Based Decision-Making 

Evidence-based decision-making is the second key design feature of the BASRC focal 
strategy. BASRC promotes this component through coaching as well as through a variety of 
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BASRC tools and processes. Instead of prescribing a set of specific classroom practices, this com- 
ponent articulates a general process for reform of instructional practice. One of the key tools in 
this process is the Cycle of Inquiry, which is a method of identifying, understanding, and meeting 
the needs of struggling students. Educators begin this process by examining student achievement 
data, including diagnostic assessments, to identify problems and determine which students are 
struggling and in which areas. BASRC also encourages schools to adopt and give more frequent 
assessments. At the school level, this cycle is designed to enable teachers to better diagnose stu- 
dents’ needs and fine-tune classroom strategies before students move on to the next grade. 
BASRC encourages districts and schools to implement the Cycle of Inquiry at many levels in the 
system (school, grade, and classroom) to identify achievement gaps, and to focus on a specific 
academic area across all levels (typically, literacy). The inquiry model assumes that districts, 
schools, and teachers will work to identify teaching practices that will boost the achievement of 
students, particularly of the lowest-performing students. 

With the goal of building districts’ and schools’ capacity to engage in evidence-based 
decision-making, BASRC uses a variety of other tools, including annual Reviews of Progress 
(in which districts and schools present evidence about progress in improving performance and 
closing achievement gaps) and the JFTK-CA database. In addition, BASRC encourages districts 
to develop systems for providing schools and teachers with useful and accessible data, and link- 
ing the data to systemwide assessment programs. 



Professional Networks and Collaboration 

Professional networks and collaboration are the third primary design feature of the 
BASRC focal strategy. This feature consists of a variety of opportunities for school leaders to 
convene and share knowledge with each other. For example, BASRC organizes professional de- 
velopment services in four different professional networks for focal district and school leaders: 

1 . the Principals’ Network for principals in all BASRC schools 

2. the Local Collaborative Coaches’ Network for school- and district-based 
coaches in all BASRC districts 

3. the District Leaders' Network for district leaders in all BASRC districts 

4. the Focal Superintendents’ Network for the superintendents in the focal dis- 
tricts only 

BASRC’ s goal for these networks is to build educators’ capacity to address student and 
school performance. Examples of topics include leadership, the Cycle of Inquiry, and equity. 
Other collaboration opportunities are BASRC-organized Summer Institutes for school and dis- 
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trict leadership teams, as well as meetings organized around particular themes, such as literacy 
and the use of data. In all of these meetings, BASRC encourages participants to continue knowl- 
edge-sharing, and develop school-level networks back at their sites, often providing them with 
tools to pass on what they have learned. 

In addition to the collaboration opportunities BASRC provides at network meetings, 
BASRC encourages districts and schools to create collaboration opportunities at their district 
offices and school sites. BASRC advocates for collaboration that supports what it refers to as 
“bottom-up and top-down” reform, in which both schools and districts share reform ideas and 
influence one another. BASRC also promotes collaboration across and within schools. The the- 
ory is that such collaboration builds knowledge and creates mutual accountability, leading to 
improved practice. 



Conclusion 

The theory of action underlying the BASRC focal reform strategy suggests that the im- 
plementation of these three design features yields changes in district, school, and classroom 
practices hypothesized to improve student outcomes, as illustrated in Figure 2.1. Taken to- 
gether, coaching, evidence-based decision-making, and networks and collaboration are thought 
to lead to districts and schools becoming more engaged in a continuous improvement process 
that is reflective, evidence-based, and collaborative. This process is supposed to help educators 
find effective ways to improve instruction for all students, and for low-performing students in 
particular, leading to improved and more equitable student outcomes. Because the focal reforms 
are process-oriented, they can result in different outcomes within each district, school, and 
classroom. The outcomes may be or may not be policies or practices that are directly linked 
with improved teaching and learning. Rather, they may be incremental improvements in culture 
that eventually foster better teaching and learning. 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 

Figure 2.1 

BASRC Focal Strategy Theory of Action 




District Reform 

Data-based decisions 
High district standards 
Evidence-based practices 
Quality professional development 
Distributed leadership 



Expected Intermediate Outcomes 



School Reform 

Data-based decisions 
Learning communities 
Knowledge sharing 
Distributed leadership 
Focus on achievement gaps 



Student Outcomes 



Improved Instruction 

Data-based decisions 
Focus on achievement gaps 
Diagnostic assessment 
Standards-based instruction 
High performance standards 




SOURCE: Adapted from the Center for Research on the Context of Teaching (CRC), Stanford University, 2004. 







Chapter 3 

Implementation of the BASRC Focal Strategy 



Chapter 2 described the theory of action behind the Bay Area School Reform Collabo- 
rative (BASRC) focal strategy, and how the strategy was intended to unfold if put into practice 
as planned. This chapter describes the places where the focal strategy was implemented and 
summarizes the lessons learned so far about actual implementation. 

Overall, the evidence presented in this chapter suggests: 

• BASRC implemented the focal strategy in a set of districts that were similar 
to, though slightly more disadvantaged than, the Bay Area as a whole. 

• Although the BASRC focal strategy originally included a model aimed at a 
set of “focal schools” within the focal districts, this school-level component 
was not implemented as planned. As a result, there was no meaningful dis- 
tinction between focal and non-focal schools in program resources or imple- 
mentation. 

• Focal districts are characterized by district-level coaching by an executive 
coach, some additional coaching or support from other BASRC staff, a sepa- 
rate BASRC-led Focal Superintendents’ Network, and more frequent review 
meetings with BASRC staff. However, important questions remain regarding 
the specific reforms undertaken as a result of these activities. 



The BASRC Focal Districts 

This report focuses on five of the six BASRC focal districts. 1 All five districts are in the 
San Francisco Bay Area. As mentioned earlier, BASRC selected the focal districts, looking in 
particular to districts where there was already a strong working relationship from Phase I and/or 
Phase II efforts. As a result, four of the six focal districts were part of earlier BASRC efforts. 
Achievement gaps and a reform-minded superintendent were also important criteria in the se- 
lection of focal districts. Within each focal district, two to five focal schools were selected 
through negotiations between BASRC and each superintendent. The schools selected may have 
been elementary, middle, or high schools, but MDRC’s research efforts have focused solely on 



'Because BASRC’s sixth focal district. Oak Grove, joined the initiative one year later and is following an 
action plan different from that of the original five districts, MDRC is not including it in the evaluation sample. 
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the elementary schools, of which there are a total of 12. Table 3.1 lists the five focal districts 
and the elementary-level focal schools within them. 



Bay Area School Reform Collaborative Focal Strategy Evaluation 

Table 3.1 

BASRC Focal Districts and Their Focal Elementary Schools 



Focal District 


Focal Elementary Schools 


Newark Unified School District 


Snow Elementary 
Musick Elementary 
Milani Elementary 


Alameda Unified School District 


Paden Elementary 
Woodstock Elementary 


Laguna Salada Union School District 


Linda Mar Elementary 
Vallemar Elementary 


San Rafael City Elementary and High 
School Districts 


Bahia Vista Elementary 
San Pedro Elementary 
Laurel Dell Elementary 


San Bruno Park Elementary School 
District 


El Crystal Elementary 
John Muir Elementary 



SOURCES: MDRC field research data. 

NOTES: Focal Districts are districts participating in the BASRC focal strategy. Focal 
Schools are schools participating in the BASRC focal strategy. 



Looking at the five BASRC focal districts in this evaluation, Table 3.2 compares the districts’ 
demographic characteristics and achievement levels before implementation of the BASRC focal 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 

Table 3.2 



Characteristics of BASRC Focal, Bay Area, and California Districts, Elementary-Level 
Average for School Years 1999-2000 Through 2001-2002 



Characteristic 


BASRC Focal 
Districts 


All Bay Area 
Districts 2 


All California 
Districts 


Demosrauhic characteristics of third-erade students 

Race/Ethnicity (%) 


White 


39.9 


51.0 


53.7 


Hispanic 


28.9 


23.6 


32.5 


Asian 


21.8 


16.0 


5.7 


Black 


5.5 


7.2 


3.8 


Other 


3.9 


2.1 


3.7 


English language learners (%) 


22.7 


18.2 


21.5 


Eligible for free or reduced-priced lunch (%) b 


29.8 


22.2 


44.6 


Achievement levels of third-erade students 

California Standards Tests 
Language arts c 


Proficient or advanced (%) 


43.1 


49.8 


35.3 


Basic (%) 


29.9 


26.2 


29.1 


Below or far below basic (%) 


26.9 


23.9 


35.5 


Mean scale score 


341.8 


349.6 


328.4 


Math d 


Proficient or advanced (%) 


49.4 


51.3 


38.7 


Basic (%) 


27.2 


24.4 


28.0 


Below or far below basic (%) 


23.6 


23.8 


33.2 


Mean scale score 


350.8 


355.8 


333.3 


Stanford Achievement Test, Ninth Edition 
Reading 


At or above 50th National Percentile Rank (%) 


59.9 


65.5 


51.8 


25th-49th National Percentile Rank (%) 


21.8 


18.2 


23.0 


Below 25th National Percentile Rank (%) 


18.3 


16.4 


25.1 


Mean scale score 


626.4 


633.5 


617.7 


Math 


At or above 50th National Percentile Rank (%) 


69.0 


71.4 


62.0 


25th-49th National Percentile Rank (%) 


18.2 


16.1 


20.2 


Below 25th National Percentile Rank (%) 


12.8 


12.4 


17.8 


Mean scale score 


620.8 


624.7 


613.0 


Sample size 


5 


106 


938 



(continued) 
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Table 3.2 (continued) 



Characteristic 


BASRC Focal 
Districts 


All Bay Area 
Districts 3 


All California 
Districts 


Demographic characteristics of fifth-grade students 

Race/Ethnicity (%) 


White 


43.1 


52.7 


55.4 


Hispanic 


26.2 


21.6 


30.6 


Asian 


21.9 


16.3 


5.7 


Black 


6.2 


7.6 


3.9 


Other 


2.5 


1.9 


3.8 


English language learners (%) 


15.5 


14.0 


17.7 


Eligible for free or reduced-priced lunch (%) b 


29.8 


22.4 


44.6 


Achievement levels of fifth-srade students 

California Standards Tests 
Language arts c 


Proficient or advanced (%) 


37.8 


47.0 


33.0 


Basic (%) 


39.8 


33.4 


38.9 


Below or far below basic (%) 


22.6 


19.6 


28.0 


Mean scale score 


338.4 


348.4 


332.0 


Math d 


Proficient or advanced (%) 


37.2 


44.0 


31.2 


Basic (%) 


32.6 


27.6 


31.4 


Below or far below basic (%) 


30.4 


28.3 


37.4 


Mean scale score 


335.6 


347.2 


325.5 


Stanford Achievement Test, Ninth Edition 
Reading 


At or above 50th National Percentile Rank (%) 


56.0 


63.9 


51.2 


25th-49th National Percentile Rank (%) 


23.4 


18.3 


22.6 


Below 25th National Percentile Rank (%) 


20.6 


17.7 


26.2 


Mean scale score 


659.1 


667.2 


654.2 


Math 


At or above 50th National Percentile Rank (%) 


63.5 


68.0 


58.0 


25th-49th National Percentile Rank (%) 


17.7 


14.7 


18.6 


Below 25th National Percentile Rank (%) 


18.7 


17.3 


23.5 


Mean scale score 


659.6 


666.9 


654.7 


Sample size 


5 


106 


944 



SOURCE: Publicly available district- and school-level data files from the California Department of Education. 
NOTES: Due to rounding, percentages may not sum to 100 percent. The estimates in the table represent averages 
across all districts, regardless of their size. 

a Bay Area districts include all districts in Alameda, Contra Costa, Marin, San Francisco, San Mateo, and Santa 
Clara counties with students in grade 3 (including BASRC focal districts). 

b Averages for free or reduced-price lunch are presented at the school level only. 

c Due to availability of test scores, California Standards Test averages in language arts are based on just two 
baseline years, 2000-2001 and 2001-2002, except for the Below or Far Below Basic average, which is based on 
one baseline year, 2001-2002. 

d Due to availability of test scores, California Standards Test averages in math are based on just one baseline 
year, 2001-2002. 
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2 

strategy with those of other districts in the San Francisco Bay Area and throughout California.” 
Achievement levels are measured by performance on both the Stanford Achievement Test, 
Ninth Edition (SAT-9), a nationally norm-referenced test, and the California Standards Tests 
(CSTs), which measure student performance relative to California state standards. The baseline, 
or preimplementation, period for the BASRC focal strategy for which MDRC has data includes 
the 1999-2000, 2000-2001, and 2001-2002 school years. 3 The baseline demographic character- 
istics and student achievement levels shown in Table 3.2 represent averages over these three 
years. Note, however, that CST scores were available only for 2000-2001 and 2001-2002 for 
reading (except for the below- or far-below-basic average, which is based on one baseline year, 
2001-2002) and only for 2001-2002 for math. 

Table 3.2 shows that, relative to Bay Area districts, 4 BASRC focal districts served a 
smaller proportion of white students and a slightly larger proportion of Hispanic and Asian stu- 
dents, but similar proportions of black students. 5 For example, on average, 40 percent of the 
third-grade students served by BASRC focal districts were white, compared with 5 1 percent in 
the Bay Area districts. At the same time, 29 percent of the third-graders in BASRC focal dis- 
tricts were Hispanic, compared with 24 percent of those in the other Bay Area districts. Table 
3.2 also shows that BASRC focal districts had slightly higher percentages of English Language 
Learners and students eligible for a free or reduced-price lunch. During the baseline period, an 
average of 23 percent of third-grade students in the focal districts were English Language 
Learners, compared with 18 percent of students in Bay Area districts as a whole. Also, 30 per- 
cent of third-graders in focal districts were eligible for the free or reduced-price lunch program, 
compared with 22 percent in the rest of the Bay Area. 6 

This table also compares characteristics of third- and fifth-grade students in the BASRC 
focal districts with students in all districts in the state of California. Contrary to the comparison 
with Bay Area districts, relative to the whole state, the BASRC focal districts included a smaller 
proportion of Hispanic students and more Asian students. BASRC focal districts also served a 
similar proportion of students who were English Language Learners. Lastly, both the focal dis- 



“Bay Area districts are defined as those districts in Alameda, Contra Costa, Marin, San Francisco, San 
Mateo, and Santa Clara counties. All districts and schools with students in these grades were included in the 
samples. 

3 Data for these and future analyses are from the California Department of Education. For more informa- 
tion on the tests, see Appendix A. 

4 This includes focal districts and districts that participated in earlier phases of BASRC reforms. 

5 The category of “other” includes Native American students and students of multiple ethnicities. The cate- 
gory “Asian” includes Pacific Islander and Filipino students. 

6 Note that the figures for eligibility for a free or reduced-price lunch represent school-level percentages 
rather than grade-level percentages. 
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tricts and the Bay Area districts as a whole served much lower proportions of students eligible 
for free or reduced-price lunch than the state average. 

Students in the focal districts generally demonstrated lower levels of reading achieve- 
ment and math achievement on the CST compared with students in the Bay Area as a whole. 
This is consistent with BASRC’s efforts to target its interventions toward relatively low- 
performing districts, particularly with respect to literacy. However, the extent to which the 
achievement levels differ varies by grade and subject. In particular, differences are larger for 
reading scores than they are for math, and the differences between BASRC focal districts and 
the Bay Area as a whole are larger among fifth-graders than they are among third-graders. This 
pattern is consistent with the literature on achievement gaps, which points to the widening of 
achievement gaps over time. The patterns of test scores on the SAT-9, also presented in Table 
3.2, seem to show similar trends. These trends suggest the possibility that the differences in stu- 
dent performances are exacerbated over time by differences in the quality of teaching and learn- 
ing in the BASRC focal schools compared with the rest of the Bay Area. In contrast, compared 
with all districts in the state of California, students in the BASRC focal districts have higher 
levels of achievement in both reading and math. 

In sum, while not all the differences are large, achievement levels in BASRC focal dis- 
tricts are lower than in other Bay Area districts for both the CST and SAT-9, and in both the 
third and fifth grades. While lower-achieving districts than these exist within the Bay Area, the 
focal districts do appear to be below average and face growing gaps as students move into 
higher grades. 



Implementation of the Core BASRC Components 

The findings in this section are based primarily on field research conducted during the 
2003-2004 school year — the second year of the BASRC focal strategy. During this year, 
MDRC visited 7 of the 12 focal elementary schools and 4 non-focal elementary schools in the 

g 

focal districts to interview principals, coaches, and teachers. MDRC also met with key BASRC 
staff and reviewed notes from interviews conducted by the Center for Research on the Context 
of Teaching (CRC) at district offices. MDRC also reviewed transcripts from interviews CRC 
researchers conducted with school, district, and BASRC staff. 

What follows is a snapshot of implementation along the three key dimensions of the 
BASRC focal strategy. This chapter describes implementation at elementary schools within the 



7 Phillips, Crouse, and Ralph (1998). 

VIDRC's findings are limited to four of the five BASRC focal districts; due to implementation difficulties 
at the fifth focal district, researchers did not have access to staff in that district. 
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focal districts (both focal and non-focal schools), and, to the extent that it is possible, implementa- 
tion at the district-office level. The next round of MDRC field work, currently underway, will fo- 
cus more attention on district-level reform activities and will aim to make connections between 
participation in the focal strategy and the reform activities found in the schools. 

So far, most of MDRC’s field research has focused on the school level. Evidence has 
been gathered during school visits regarding the reform activities along the key dimensions of 
the BASRC focal strategy: coaching, evidence-based decision-making, and networks and col- 
laboration. Important questions remain regarding the extent to which many of these activities 
are the result of the district’s participation in the focal strategy as opposed to earlier BASRC 
reforms or ongoing district initiatives. It would not be unusual to see some of these activities in 
schools that participated in earlier BASRC initiatives (Phase I and Phase II), but it is unclear 
whether these schools have been more successful in implementing these activities because they 
are in focal districts. As field research continues, particularly at the district level, MDRC will try 
to better understand the connection between participation in the focal district strategy and im- 
plementation of school-level reforms. 

Coaching 

The following section summarizes key observations regarding the implementation of 
coaching in the second year of the BASRC focal strategy: 

• In each focal district, BASRC assigned an executive coach to work with the 
superintendent. All of the executive coaches served as advisors according to 
the needs of each district. 

In the 2002-2003 school year, BASRC assigned executive coaches to each BASRC fo- 
cal district. CRC interviews with executive coaches and MDRC interviews with BASRC staff 
coaches indicated that executive coaches aimed to help superintendents define their roles and 
improve communication with central office colleagues, the school board, and school-level staff. 
Executive coaches also helped address a range of system reform efforts, which were either iden- 
tified through BASRC’ s needs assessment or decided on before the onset of the focal strategy. 
BASRC’ s goals for executive coaching included promoting academic coherence (for example, 
alignment between the reading curricula and teacher professional development), creating ac- 
countability structures, and advocating for the implementation of data-reporting systems. How- 
ever, the extent to which coaches actually delved into these issues is not yet known. The avail- 
able data suggest that coaching focused mostly on building capacity for reform by strengthening 
leadership, developing a focus for reform work, and improving communication. 

• In Year 2, there was very little coaching by BASRC staff at the focal schools. 
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At the outset of the BASRC focal reform strategy, BASRC planned to include school- 
level coaching by BASRC staff as an important part of the focal reforms. In Year 1 (the 2002- 
2003 school year), school-level coaches focused on engagement and assessing the needs of their 
schools. For example, they made sure there was a school leadership team in place, met with 
principals to decide which issues to work on, and defined roles for reform efforts. However, by 
the following year, many of the school-level BASRC coaching relationships faded, and the 
BASRC staff in this role instead focused on district-level reform efforts. 9 This shift took place 
for various reasons, including: resistance from principals and other school staff, redundancy 
with the roles of Local Collaborative coaches, and a shift in BASRC’s focus. The BASRC staff 
coaches did make themselves available as resources for principals and other school leaders at 
focal and non-focal schools. However, rather than supporting individual school reforms, they 
generally supported the executive coaches and worked with district staff, principals, and school 
leaders on reform efforts across schools. For example, in one district, a BASRC staff coach 
working on district-level reforms met with all the principals in the district to communicate in- 
formation about a curriculum adoption. By Year 2, because there was no school coaching effort 
specifically aimed at most of the focal schools, there was little distinction between “focal” and 
“non-focal” schools. 

• Within the school system, Local Collaborative (LC) coaches promoted bot- 
tom-up and top-down reform work, engaging with district staff, principals, 
and literacy coaches at the school or cross-school level. 

Local Collaborative coaches were district or school employees who support BASRC re- 
forms within and across schools. Three of the four focal districts visited by MDRC had one or 
more Local Collaborative coaches working within or across schools. Though these efforts were 
ongoing in the BASRC focal districts, they were not a part of the focal strategy per se, but a con- 
tinuation of earlier BASRC reform efforts. The LC coaches were chosen from among individuals 
playing a variety of roles in their district. They were teachers, existing school- or district-based 
coaches, principals, or district administrators. BASRC grants were used to give educators some 
release time to serve as LC coaches. The responsibilities of the LC coaches varied, but they gener- 
ally worked with school-based literacy coaches and/or leadership teams to guide reform efforts. 
They helped to facilitate grade-level or school-level meetings and they often conveyed informa- 
tion from the district office and spread ideas from other schools in the district. For the most part, 
BASRC trained these coaches through the Local Collaborative Coaches Network (a feature of the 
Phase II strategy), or during Phase I activities. By Year 2, most of these coaches were already 
working either within or across schools, regardless of their focal or non-focal status. In Year 2, LC 
coaches continued to attend the network meetings, picking up new tools and activities, and could 



9 Coggins (2004). 
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utilize BASRC staff as a resource. They met with schools across the collaborative or district, set- 
ting up monthly meetings that included some of the following activities: sharing BASRC tools 
and rubrics; preparing and discussing assessment data; and facilitating discussions around a par- 
ticular school, strategic plan, or other facet of the reform effort. 

• At both focal and non-focal schools, literacy coaches, reform coordinators, 
and other school leaders served as school-level coaches, supporting the cur- 
riculum adoptions and reform work of the school and district. 

Again, as part of earlier BASRC reforms, in all four focal districts, most of the focal 
and non-focal schools MDRC visited had either a part-time or full-time literacy coach. These 
positions were sometimes funded by BASRC grants, and sometimes through other resources. 
Nevertheless, the work of these coaches was typically consistent with or influenced by the 
school’s BASRC reform work (for example, inquiry around literacy instruction). These coaches 
helped teachers review assessment data and led discussions around curriculum implementation 
at the grade, school, or classroom level. In the second year of implementation, in schools that 
had already adopted the state-required literacy programs, the literacy coaches and mentor teach- 
ers focused much of their attention on the implementation of the programs. They worked with 
teachers on lesson planning and pacing guides, occasionally modeling instruction and pulling 
out struggling students. Often, these school-based coaches collaborated with their Local Col- 
laborative coaches and with coaches from other schools. 

• BASRC supplied tools and resources to district administrators and school- 
based coaches through its formal networks and through BASRC coaches. 

Wi t hin the focal districts, coaches used BASRC tools in several ways. In at least one dis- 
trict, the BASRC school coaches along with the principals reviewed Just for the Kids - California 
reports of CST data in order to set targets for student achievement. In several focal and non-focal 
schools, the Local Collaborative coach or school leaders learned about activities and collected 
meeting ideas from BASRC networks or coaches, and brought them back to use in meetings in 
their district. Likewise, these coaches used Cycle of Inquiry worksheets and diagrams that they 
received to explain inquiry and also supply a way for teachers to record and track information. In 
some instances, coaches recreated the documents, changing some of the language and removing 
the BASRC logo, in order to make teachers feel that the process was more a part of their own 
work. 



Evidence-Based Decision-Making 

The following section presents key findings about the implementation of evidence- 
based decision-making in Year 2 of the BASRC focal strategy: 
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• At least three of the four focal districts required assessments measuring pro- 
gress in language arts and/or math several times during the school year, but 
several obstacles impeded the extent to which these assessments were sys- 
tematically made available and used in district- or school-level decision- 
making. 

In 2003-2004, the second year of the BASRC focal reform strategy, at least three of the 
four districts administered districtwide assessments and collected the test score data several 
times a year. MDRC still needs to learn more about the extent to which BASRC influenced fo- 
cal districts in how they chose or used assessments. With the adoption of state-required literacy 
curricula (Houghton-Mifflin or Open Court), which include frequent assessments, the number 
of measures districts collect has been growing. Each of the districts had a vision for how the 
required assessments could inform practice at various levels in the system. However, a variety 
of obstacles seemed to have impeded implementation. For example, one district’s assessments 
were frequently changed, making it difficult to track trends over time. In another district, teach- 
ers reported a lack of adequate professional development to help them analyze assessment data 
or realize the value of the analysis. Moreover, limited access by schools and teachers to assess- 
ment data was an ongoing obstacle in all districts. This is discussed further below. 

• In Year 2 of the BASRC focal reforms, at least two of the four focal districts fo- 
cused on making assessment data more readily accessible to schools and teach- 
ers; the efforts looked promising but the systems were still in development. 

Two of the focal districts were developing data systems that would make classroom- 
and school-level data accessible to teachers and principals. In one district, the system is com- 
prehensive and customized to the districts’ instructional goals. It contains, for example, all 
Houghton-Mifflin assessments, with state standards crosswalked with test questions. Teachers 
will ultimately have the ability to compile information on a student across assessments accord- 
ing to a particular standard. As of Year 2, this district was in the very early stages of implemen- 
tation, and the data systems were not yet in widespread use. A second district also had plans to 
make assessment data available to teachers through a networked system, but had not yet speci- 
fied an actual plan. The other two districts relied on district staff to produce hard copy charts for 
school site leaders. MDRC does not know the frequency, detail, and usefulness of their ap- 
proach; such details represent important areas for inquiry as MDRC’s study progresses. The 
focus on improving data systems may have been aided by the support of an executive coach, but 
these efforts typically began before the focal strategy was implementated. 

• Among both the focal and non-focal school sites, the most widely mentioned 
BASRC concept for evidenced-based decision-making was a focus on “tar- 
get students.” 
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The strategy of selecting a few “target students” in each classroom to whom teachers 
pay particular attention in order to evaluate the effects of their practice was an idea proposed by 
BASRC and disseminated through network meetings and coaching. It seems to have been a 
popular idea that was readily adopted at most BASRC schools that MDRC visited. In two of the 
districts, the target student strategy was supported or required by the district central office. 
However, all of the schools visited referred to the selection of and attention to “target students.” 
Typically, every teacher in the school was encouraged or required to select two to four target 
students each fall, based on report cards, CSTs, other assessments, or anecdotal data. Most 
schools focused on the lowest-performing students; at least one school advised teachers to 
choose a mix of levels; and others chose low-performing students who they thought had the best 
chance of advancing. Throughout the year, teachers discussed target students in grade- or 
school-level meetings or in meetings with the principal. Also, teachers were often encouraged 
by school leaders to individually consider the needs of their target students, to consider inter- 
ventions or changes in practice appropriate for them, and to assess the results of their changes. 
The practice of identifying and working with “target students” originated in earlier BASRC 
work, and precedes the BASRC focal strategy. 

• BASRC’ s “brand” of inquiry was modeled for focal districts and schools 
through quarterly review meetings and through an annual Review of Pro- 
gress meeting facilitated by BASRC staff. The former was unique to the fo- 
cal strategy, while the latter was part of the larger Phase II strategy. 

Two BASRC accountability exercises — quarterly review meetings and annual Review 
of Progress meetings — provided opportunities for BASRC coaches and staff to model the in- 
quiry practices they espoused. At these meetings, BASRC staff led district and school leaders 
through an exercise of analysis and reflection, providing an example of the types of practices 
they encouraged in district offices and school sites. These meetings therefore served not only as 
a review of districts’ progress on reform efforts and in achievement trends, but also as a way to 
deliver a key feature of the focal strategy. Leaders in focal districts attended more review meet- 
ings. As a result, they may have become more skilled at or more inclined to adopt inquiry prac- 
tices. However, the extent to which the focal districts actually implemented these inquiry prac- 
tices is not clear. This is an important area that MDRC will study as the evaluation proceeds. 
The extent to which inquiry was conducted at the school level is discussed below. 

• Across all four districts, most of the schools reported conducting BASRC’ s 
Cycle of Inquiry or practicing “inquiry,” but “inquiry” represented a different 
set of activities at each school. 

For all the schools, inquiry seemed to be interpreted broadly as analysis and reflection. 
The activities school leaders and teachers described to illustrate their inquiry practices ranged 
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from informal conversations about student work to filling out detailed worksheets with ques- 
tions about student assessment data. They typically cited collaborative situations when describ- 
ing inquiry, either regular grade-level meetings or casual conversations with colleagues. Teach- 
ers said they also practiced inquiry at the classroom level, which usually seemed to be equated 
with attention to target students. The teachers seemed supportive of inquiry practices as long as 
they remained informal. There was “pushback” when they were asked to fill out forms. In addi- 
tion, inquiry at the school level was strongly encouraged by one of the focal districts. In Year 2, 
this district required that schools follow a Cycle of Inquiry protocol when developing their an- 
nual school improvement plans for the district. In general, schools in the other focal districts did 
not report the presence of similar types of district-level inquiry requirements. 

• The adoption of the Houghton-Mifflin or Open Court reading programs was 
often seen as a conflict with target student and inquiry practices. 

In the 2003-2004 school year, Year 2 of the focal strategy, schools were implementing 
or preparing to implement the state-required reading program selected by their district. All four 
of the districts were requiring high fidelity in the adoption of the reading program. Typically, 
this meant that teachers (at least most teachers) were expected to use the program’s teaching 
materials in their lessons, administer the program’s assessments selected by their district, and 
stick to a schedule planned by their district. Many school leaders, coaches, and teachers viewed 
the requirements of the reading program adoption as a contradiction with or obstacle to conduct- 
ing inquiry. They said that this structure did not allow for analysis or reflection and that the 
complexity of learning a new program took over all collaboration and professional development 
time. As MDRC conducts additional district-level research, it will be important to ascertain how 
the district central offices perceived these changes, and if they too (as well as BASRC staff) saw 
them as running counter to the Cycle of Inquiry or other BASRC focal reforms. 

Networks and Collaboration 

As mentioned in Chapter 2, BASRC organized and led regular meetings for principals, 
Local Collaborative coaches, and district administrators, to provide networking opportunities 
and share ideas. BASRC also promoted networking and collaboration as a key to sharing reform 
ideas within and across schools. This section summarizes key observations regarding networks 
and collaboration in Year 2 of the focal strategy, 

• District leaders, coaches, and principals in the focal districts had the opportu- 
nity to attend meetings of networks organized by BASRC: the Focal Superin- 



10 These findings will be reported in MDRC’s next BASRC report. 
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tendents’ Network, the Local Collaborative Coaches’ Network, and the Prin- 
cipals’ Network. The extent of participation in these meetings varied. 

When interviewing coaches and principals, MDRC asked whether they attended the 
BASRC network meetings, and if so, what they got out of the meetings. Local Collaborative 
coaches, and sometimes other school coaches (reform coordinators or literacy coaches) attended 
the BASRC Local Collaborative Coaches’ Network meetings pretty consistently. They reported 
that they came away with tools and activities that they could use back at their schools. Some 
principals attended every Principals’ Network meeting, while others attended sporadically or 
not at all, citing scheduling difficulties. Those who attended said they welcomed the opportunity 
to talk with other principals, especially those from other districts, and occasionally were moti- 
vated by guest speakers. However, the principals interviewed did not provide any examples of 
specific benefits, particularly pertaining to instructional reform at their schools. As with the 
other BASRC components, participation in these coach and principal networks did not seem to 
vary by focal school status. 

• In the focal districts, it was common for school leaders (principals and/or teacher 
leaders and coaches) at both focal and non-focal BASRC schools to meet with 
each other on a regular basis to share ideas and advance their reform goals. 

These meetings were part of the “Local Collaborative,” a structure introduced as part of 
the larger Phase II work and which included a group of BASRC Phase Il-funded schools and 
the district office. In one focal district, the BASRC schools were grouped into three Local Col- 
laboratives, but more typically, each district had one Local Collaborative composed of all the 
BASRC grantees. The structure was intended to promote collaboration and both bottom-up and 
top-down spread of reform ideas. The frequency, attendance, and content of LC meetings varied 
by Local Collaborative. Some groups convened monthly; others struggled with poor attendance 
and met only sporadically. Some teams discussed current reform issues — often related to the 
new or impending adoption of one of the state-required reading programs. Some teams planned 
joint professional development activities or shared guest speakers for staff or parents. In one 
district, the team developed a districtwide survey regarding target students. The absence of dis- 
trict leaders at some LC meetings sometimes created misunderstandings. In one district, school 
leaders thought district administrators were no longer focused on BASRC reforms, not realizing 
that recent district initiatives were part of the BASRC focal reforms that evolved from the col- 
laboration of district leaders with a BASRC executive coach. 

• Other than districtwide professional development sessions, there were few 
opportunities for teachers to collaborate across schools. 

There were a few examples of cross-site collaboration, but for the most part, collabora- 
tion across schools was typically limited to school leaders. In one district, teachers in both focal 
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and non-focal schools had the opportunity to attend collaborative professional development 
workshops on cultural awareness and English language development (ELD) strategies. In an- 
other district, teachers from one school modeled a lesson from a new writing curriculum for 
teachers at another school that was gearing up to implement the same program. 

• At schools in the focal districts, both planned and informal collaboration was 
widespread. 

Most BASRC schools had grade-level collaboration time built into their schedules, of- 
ten paid for with BASRC grants. Teachers used the time to discuss concerns about students, 
offer ideas for teaching strategies, and go over student assessment data or student work. In some 
meetings, teachers participated in inquiry practices as described above, with guidance from a 
coach or reform coordinator. However, at schools in the first year of adopting one of the state- 
required reading programs, teachers most often used the time to share challenges and ideas re- 
lated to the reading program. Collaboration in the form of shared teaching or teaching observa- 
tions was much less common. While there were a few attempts in a handful of schools, teachers 
resisted these sorts of activities, primarily because they were reluctant to leave their classrooms 
and were uncomfortable with peer evaluations. 



Summary 

Beginning in the 2002-2003 school year, the BASRC focal strategy was put in place in 
several districts that were relatively similar to the rest of the San Francisco Bay Area. To the 
extent that they were different, they served fewer white students and students with lower aver- 
age achievement. 

In terms of implementation of the focal strategy, the primary features that distinguished 
BASRC focal districts from non-focal districts appear to be: district-level coaching by an execu- 
tive coach, sometimes additional coaching or support from other BASRC staff, a separate 
BASRC-led Focal Superintendents’ Network, and more frequent review meetings with BASRC 
staff. However, important questions remain regarding the specific policies, programs and strate- 
gies undertaken as a result of these activities. In focal districts, did coaching activities or interac- 
tions with BASRC staff lead to reform goals or reform activities? How are reform activities in 
focal districts different from those in non-focal districts? What is the anticipated path from re- 
form activities to improvements in student achievement? Going forward, MDRC’s field work 
will explore these questions. If the focal districts are not engaging in reforms in a way that is 
more substantial than or different from the reforms in comparable districts, then changes in stu- 
dent achievement above and beyond those seen in similar districts are unlikely. 
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At the school level, there do not appear to be differences between the reforms occurring 
in focal and non-focal schools. By Year 2, the coaching by BASRC staff, which was originally 
planned to distinguish focal schools from non-focal schools, was not in place in most of the fo- 
cal schools visited by MDRC. While some focal schools did interact with BASRC staff 
coaches, the interactions were limited, and no more extensive than non-focal schools’ interac- 
tions with the same BASRC personnel. At both focal and non-focal schools, MDRC did ob- 
serve evidence of all three of the key features of the BASRC focal reform strategy (coaching by 
district or school staff, evidenced-based decision-making, and networks/collaboration). How- 
ever, given that these features are also central to BASRC’ s Phase I and Phase II strategies, it is 
possible that these were implemented as a result of participation in earlier BASRC reform ef- 
forts rather than the BASRC focal strategy. Moreover, it is also possible that these reform prac- 
tices were in place before any participation in BASRC reform efforts. As such, the evolution of 
these reforms in schools within the BASRC focal districts is an important part of MDRC’s in- 
quiry in its current field work and will be described in the next report. 
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Chapter 4 



The Relationship Between Participation in the BASRC 
Focal Strategy and Trends in Student Achievement 

The ultimate aim of the BASRC focal strategy is to raise achievement among all students 
and accelerate achievement gains among the lowest-performing students. This chapter addresses 
the question of whether participation in the BASRC focal strategy was associated with district- 
wide improvements in student achievement — above and beyond the progress that would have 
occurred otherwise — in the five Bay Area school districts in which it was implemented. To ac- 
complish this, the chapter assesses the degree of academic progress in these BASRC focal dis- 
tricts, and whether or not it outpaces progress in similar districts that did not participate in the 
BASRC focal strategy. In particular, this chapter answers the following questions: 

• How does student performance in the BASRC focal districts compare with 
student performance in those districts prior to the implementation of the focal 
strategy? 

• How do these changes in student performance in the BASRC focal districts 
compare with the changes in student performance in similar districts in the 
San Francisco Bay Area? 

Below is a brief overview of the findings, discussed in detail later in the chapter. 

• In the years following implementation of the BASRC focal strategy, third- 
grade students in the BASRC focal districts and in similar districts through- 
out the Bay Area showed no progress in student achievement compared with 
the baseline period. In fact, average proficiency rates declined. 

• On the other hand, fifth-grade students’ performance in the focal districts im- 
proved over time, slightly outpacing improvements in the comparison dis- 
tricts, but the differences were not statistically significant. 

• Among blacks and Hispanics, English Language Learners, and economically 
disadvantaged students, reductions in the percentage of low-performing fifth- 
grade students in the BASRC focal districts in Year 2 outpaced the reduc- 
tions in low-performing fifth-graders in the comparison districts. However, 
these differences were not statistically significant. 

The section below provides a description of MDRC’s analytic approach to answering 
the questions outlined above, followed by an explanation of the baseline characteristics and per- 



33 




formance levels in the BASRC focal districts and their comparison counterparts. Finally, the 
findings are discussed in detail, presenting preliminary results for the first two years of the 
BASRC focal strategy. 



The Analytic Approach 

The most challenging aspect of assessing whether an intervention improves student per- 
formance outcomes is determining what would have happened if the intervention had not been 
adopted. This unobservable scenario is referred to as the “counterfactual.” Only by estimating 
the counterfactual can the “impact” or “net effect” of a reform be understood. Random assign- 
ment of students or schools to a reform program is the most reliable basis from which to esti- 
mate the counterfactual. When random assignment is not a feasible option, often the next best 
approach is to combine an “interrupted time series” (ITS) or “difference in differences” ap- 
proach with comparison groups. 

The ITS approach posits that, absent any change in student population, the best predic- 
tor of future educational outcomes in the given district or school is past educational outcomes 
for that same entity . 1 However, a simple comparison over time does not account for the possibil- 
ity that local events (for example, changes in state policy) not related to the implementation of 
the reforms in question are driving any observed progress. In order to account for this “local 
history” as much as possible, MDRC compares changes over time in the set of BASRC focal 
district with changes over time in a set of similar districts from the same local context. This 
analysis relies on three basic comparisons: 

• The difference between the preimplementation (baseline) average of student 
outcomes in BASRC focal districts and actual student outcomes in the years 
after implementation of the BASRC focal reform strategy (that is, the devia- 
tion from baseline). 

• The difference between average baseline achievement and actual student 
achievement over the same period of time in a set of carefully chosen com- 
parison districts. 

• The difference between changes over time in achievement at the BASRC fo- 
cal districts versus changes over the same period of time in their comparison 
counterparts (that is, the difference in the deviation from baseline). 



'in other studies, this logic has been applied to schools. Examples of those studies include Bloom (1999 
and 2003), Bloom, Ham, Melton, and O'Brien (2001), Snipes (2003), and Kernple, Herlihy, and Smith (2005). 
In the case of BASRC, similar logic is applied to disUicts. 
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If a reliable counterfactual can be estimated, applying the comparative ITS approach 
can provide a reliable estimate of the net effect of a reform. In previous evaluations, MDRC has 
used comparative ITS methods to distinguish between the effects of particular programs and 
progress that would have been observed without the reforms in question . 2 The evaluation of 
BASRC differs from these other studies in that, due to the nature of the BASRC focal strategy 
and the manner in which it evolved, it is difficult to develop a reliable estimate of the outcomes 
that would have been observed in the absence of the strategy. Therefore, in this particular case, 
while the results of the analysis can indicate whether there is an association or correlation be- 
tween the BASRC focal strategy and changes in student achievement, they do not provide a 
sufficient basis for ascertaining whether there is in fact a causal relationship between the two. 

In particular, BASRC staff tried to select “reform-minded” districts for participation in the 
focal strategy. These districts may differ from the comparison districts in ways that are neither 
quantifiable nor observable. It is therefore possible that — even in the absence of the BASRC fo- 
cal reforms — these districts were more likely to implement reforms and improve student per- 
formance than districts with similar student populations and achievement track records. This pos- 
sible predisposition is exacerbated by the presence of earlier phases of BASRC reforms not only 
in the BASRC focal reform districts, but also in some of the comparison districts . 3 As a result, the 
“interruption” in time, as well as the contrast between the focal reform and comparison districts, is 
more difficult to interpret than in other evaluations. Finally, the fact that the focal strategy (and 
therefore the analysis) is implemented at the district level as opposed to the school level limits the 
sample size and reduces the power of any statistical inferences . 4 

Nonetheless, the analysis presented in this chapter sheds light on whether student 
achievement outcomes in BASRC focal districts changed in ways that are systematically differ- 
ent from those of similar districts in the Bay Area. In other words, though it is uncertain whether 
or not the BASRC focal strategy caused changes in student achievement, MDRC can at least 
explore the extent to which the BASRC focal strategy is associated with improved achievement 
outcomes. While this does not isolate the effect of the BASRC focal reforms, it may aid the dis- 
cussion of the BASRC focal reform strategy’s viability as a means of improving student 
achievement . 5 



2 Kemple, Herlihy, and Smith (2005), Snipes (2003), and Bloom (1999, 2003). 

3 Unfortunately, it was not possible to rule out Bay Area districts that participated in earlier phases of 
BASRC and maintain a sufficient sample size for the study. 

4 It is important to note that the district-level analysis might limit statistical power in two ways. First, focus- 
ing on districts limits the number of observations that can be included in the study. Second, most variation in 
achievement is within rather than across districts, which will drive up the standard errors and reduce the preci- 
sion of the estimates. 

5 A potential threat to the validity of the comparative ITS approach is the possibility of systematic changes 
over time in the student populations in the BASRC focal districts or their comparison counterparts (for exam- 

(continued) 
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Comparison Districts for the BASRC Focal Districts 

As discussed above, including comparison districts in the analysis helps provide an es- 
timate of the progress that would have occurred in the BASRC focal districts in the absence of 
the BASRC focal reforms. The approach helps account for local or regional factors above the 
district level that may influence student performance, such as a change in state policy or state 
tests. To execute this strategy, each BASRC focal district was matched with a set of similar 
non-focal Bay Area districts. 6 

Logic suggests that the most accurate predictor of future performance on a particular out- 
come is past performance on that same outcome. Since the focus of this analysis is BASRC’ s rela- 
tionship to elementary-student achievement, prior academic achievement among elementary stu- 
dents in these districts was the primary criteria upon which comparison districts were selected. 
Given BASRC’s focus on literacy, MDRC focused in particular on student performance on read- 
ing or language arts assessments. It is possible that, even among districts with similar prior 
achievement, districts that serve different student populations might be expected to evolve differ- 
ently over time. Therefore, MDRC also matched districts in terms of racial/ethnic composition. 
Finally, districts were also matched on their size. 7 In order to capture schools’ and districts’ influ- 
ence on students by the end of elementary school, all matching was done at the fifth-grade level. 

The BASRC focal strategy was implemented in the five focal districts in this study dur- 
ing the 2002-2003 school year. Therefore, the baseline period for this analysis is the three years 
immediately preceding this point, the 1999-2000 through 2001-2002 school years. Comparison 
districts were chosen on the basis of their similarities throughout this baseline period. 8 Appendix 
B contains a more detailed description of the comparison-district selection process. 

The matching process resulted in a set of comparison districts with similar baseline 
achievement patterns and student populations. The five focal districts were matched with 15 non- 



pie, demographic changes in the surrounding community or districts changing their geographic boundaries). It 
is also possible that the adoption of a reform program can cause a change in the student population by, for ex- 
ample, increasing attendance and reducing mobility. MDRC’s analysis of demographic shifts suggests that 
there weren't significant systematic shifts in the demographics of either the focal or the non-focal districts dur- 
ing the baseline or follow-up periods of the analysis. Therefore, this chapter presents results that do not control 
for changes in district composition. 

6 The San Francisco Bay Area was defined to include six counties: Alameda, Contra Costa, Marin, San 
Francisco, San Mateo, and Santa Clara. 

7 This was done by matching on the number of fifth-grade students in the district. 

8 The preimplementation or baseline period from which MDRC determines the historical patterns varies 
for different outcomes due to data availability. When assessing baseline patterns on the nationally norm- 
referenced test, the SAT-9, this study relies on three years of baseline data: 1999-2000 through 2001-2002. 
When assessing baseline patterns on the newer test in California, the California Standards Test (CST), the 
study relies on just two years of baseline data to project future trends: 2000-2001 and 2001-2002. 



36 




focal districts, with some comparison districts matching with more than one focal district. Overall, 
as shown in Table 4.1, the two sets of districts look very similar, particularly on the characteristics 
used for matching. For example, across all baseline years, the focal and comparison districts 
matched within 2 percentage points on the California Standards Test (CST) proficiency levels 
(proficient/advanced, basic, and below/far below basic) for both third- and fifth-grade students. 
The focal districts and the comparison districts also look similar in terms of students’ 
race/ethnicity. For example, in the focal districts, 26 percent of fifth-grade students were Hispanic, 
compared with 27 percent in the comparison districts; 22 percent of the students in the focal dis- 
tricts were Asian, compared with 17 percent in the comparisons; and 6 percent of the students in 
the focal districts were black, compared with approximately 5 percent in the comparisons. 

The two sets of districts are also comparable in characteristics that were not the basis for 
matching. Table 4.1 also shows that the comparison districts are within 5 percentage points of the 
focal districts in the percentage of English Language Learners (approximately 16 percent versus 
14 percent), the percentage of students eligible for free or reduced-price lunch (30 percent versus 
approximately 25 percent), and on performance levels on the math portions of the CST and Stan- 
ford Achievement Test, Ninth Edition (SAT-9) for both third-grade and fifth-grade students. 

The most apparent difference between the focal districts and comparison districts is in 
the average number of third- and fifth-grade students enrolled in each district. The comparison 
districts are on average larger than the focal districts (599 fifth-grade students compared with 
499 in the comparisons), but had fewer schools (six elementary schools per district compared 
with eight in the focal districts). Together, these two indicators suggest that the comparison dis- 
tricts had more classes per school (or more students per class). That is, they likely had larger 
schools than the focal districts. 



Findings 

This section presents preliminary findings on the relationship between the BASRC fo- 
cal strategy and changes in student achievement in the first two years of the strategy (2002-2003 
and 2003-2004). These findings are presented along with figures that compare the average 
achievement levels of the five BASRC focal districts with those of the non-focal comparison 
districts. Lor each group, averages are presented for third-grade and fifth-grade students during 
the two or three years before the introduction of the BASRC focal strategy (the baseline period), 
and for each of the two years after the focal strategy was adopted (the follow-up years). 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 

Table 4.1 



Characteristics of BASRC Focal and Comparison Districts, Elementary-Level 
Average for School Years 1999-2000 Through 2001-2002 





BASRC Focal 


Comparison 


Characteristic 


Districts 


Districts 



Demographic characteristics of third-m ade students 



Average number of third-grade students 


505 


601 


Average number of elementary schools 3 


8 


6 


Race/Ethnicity (%) 


White 


39.9 


47.7 


Hispanic 


28.9 


28.8 


Asian 


21.8 


17.4 


Black 


5.5 


4.8 


Other 


3.9 


1.3 


English language learners (%) 


22.7 


18.2 


Eligible for free or reduced-priced lunch (%) b 


29.8 


24.7 


Achievement levels of third-grade students 

California Standards Test 
Language arts c 


Proficient or advanced (%) 


43.1 


41.2 


Basic (%) 


29.9 


30.9 


Below or far below basic (%) 


26.9 


28.0 


Mean scale score 


341.8 


339.3 


Math d 


Proficient or advanced (%) 


49.4 


46.2 


Basic (%) 


27.2 


26.8 


Below or far below basic (%) 


23.6 


26.8 


Mean scale score 


350.8 


346.1 


Stanford Achievement Test, Ninth Edition 
Reading 


At or above 50th National Percentile Rank (%) 


59.9 


59.5 


25th National Percentile Rank (%) 


21.8 


22.4 


Below 25th National Percentile Rank (%) 


18.3 


18.2 


Mean scale score 


626.4 


624.7 


Math 


At or above 50th National Percentile Rank (%) 


69.0 


68.1 


25th National Percentile Rank (%) 


18.2 


18.2 


Below 25th National Percentile Rank (%) 


12.8 


13.7 


Mean scale score 


620.8 


618.6 


Total number of districts 


5 


15 



(continued) 
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Table 4.1 (continued) 





BASRC Focal 


Comparison 


Characteristic 


Districts 


Districts 



Demographic characteristics of fifth-grade students 



Average number of fifth-grade students 


499 


599 


Average number of elementary schools 3 


8 


6 


Race/Ethnicity (%) 


White 


43.1 


49.2 


Hispanic 


26.2 


26.8 


Asian 


21.9 


17.4 


Black 


6.2 


5.3 


Other 


2.5 


1.3 


English language learners (%) 


15.5 


14.0 


Eligible for free or reduced-priced lunch (%) b 


29.8 


24.7 


Achievement levels of fifth-grade students 

California Standards Test 
Language arts c 


Proficient or advanced (%) 


37.8 


38.6 


Basic (%) 


39.8 


41.1 


Below or far below basic (%) 


22.6 


20.3 


Mean scale score 


338.4 


341.1 


Math d 


Proficient or advanced (%) 


37.2 


36.9 


Basic (%) 


32.6 


33.5 


Below or far below basic (%) 


30.4 


29.6 


Mean scale score 


335.6 


335.7 


Stanford Achievement Test, Ninth Edition 
Reading 


At or above 50th National Percentile Rank (%) 


56.0 


57.9 


25th National Percentile Rank (%) 


23.4 


23.1 


Below 25th National Percentile Rank (%) 


20.6 


19.0 


Mean scale score 


659.1 


659.7 


Math 


At or above 50th National Percentile Rank (%) 


63.5 


63.5 


25th National Percentile Rank (%) 


17.7 


17.9 


Below 25th National Percentile Rank (%) 


18.7 


18.6 


Mean scale score 


659.6 


659.4 


Total number of districts 


5 


15 



(continued) 
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Table 4.1 (continued) 



SOURCE: MDRC calculations from district and school records from the California Department of 
Education. 

NOTES: Comparison district columns represent the average of the average across each comparison 
district cluster. Note that some comparison districts match with more than one focal district. The 
estimates in the table represent averages across all districts, regardless of their size. 

'Elementary schools include any school with a fifth-grade class, since comparison districts were 
selected from fifth-grade outcomes. 

b Averages for free or reduced-price lunch are presented at the school level only. 

c Due to availability of test scores, California Standards Test averages in language arts are based on 
just two baseline years, 2000-2001 and 2001-2002, except for the Below or Far Below Basic average, 
which is based on one baseline year, 2001-2002. 

J Due to availability of test scores, California Standards Test averages in math are based on just one 
baseline year, 2001-2002. 



MDRC has focused its analyses on third- and fifth-grade students for two reasons. First, 
test scores of third-graders are more reliable than those of second-graders, but still capture re- 
sults of students who are learning early reading skills . 9 Much of the current accountability ef- 
forts are focused on whether or not students can read by the third grade. Fifth-graders allow for 
an analysis of a slightly older group of students who have been in school longer, are learning 
more advanced reading skills, and reflect a more cumulative effect of the quality of education 
across the elementary grades. Second, achievement gaps tend to be larger in later grades, so 
fifth grade provides a good opportunity to see if efforts to support the lower achievers close the 
performance gap. The sections below present findings for all third- and fifth-grade students and 
then examine the trends of key subgroups, including minority students, English Language 
Learners, and economically disadvantaged students. 

Presented first are findings for performance on the language arts portion of the CST. 
The CST is a high-stakes test in California, as state accountability requirements and sanctions 
hinge on schools’ CST results. Next, this section presents findings regarding students’ perform- 
ance relative to national norms on the state’s nationally norm-referenced achievement tests . 10 



9 Shepard, Kagan, and Wurtz (1998). 

10 In particular, the state administered the Stanford Achievement Test (SAT-9) during the years immedi- 
ately preceding implementation of the BASRC focal strategy, and the California Achievement Test (CAT-6) in 
the years following implementation. Though the test changed, several measures that remain constant over time 
are still available. These include the percentage of students who performed at or above the 50th percentile and 
the percentage of students who fell short of the 25th percentile. 
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Findings for All Students 

The following summarizes the findings from the analysis of the relationship between 
BASRC focal district status and progress in student achievement measures over the first two 
years of the intervention: 

• Achievement among third-grade students in BASRC focal and comparison 
districts, as measured by performance in language arts on the CST, declined 
in the two years after implementation of the BASRC focal strategy. Though 
not statistically significant, the declines were greater in the BASRC focal dis- 
tricts than in their non-focal counterparts throughout the Bay Area. 

• The percentage of fifth-grade students performing at proficient or advanced 
levels in language arts on the CST increased during the first two years of the 
BASRC focal strategy. This pattern was mirrored by the changes in achieve- 
ment observed in the comparison districts in the Bay Area. 

• The percentage of fifth-grade students scoring below basic in language arts 
on the CST fell over the course of the first two years of the BASRC focal re- 
form strategy, indicating improvements among low-performing students. 
Similar improvements did not appear to take place in comparison districts in 
the Bay Area. However, these differences are not statistically significant, and 
may be due to chance. 

• For both third- and fifth-grade students, performance on nationally norm- 
referenced tests suggests that scores at BASRC focal schools, as well as in 
the Bay Area in general, fell when the state changed assessments in the first 
year of the focal reforms. Among fifth-graders, this decline seemed some- 
what smaller in the BASRC focal districts than in comparison districts from 
the same area. However, as is the case with several other findings, the differ- 
ences are not statistically significant. 



Student Performance on the CST 

Figure 4.1 shows the average percentage of third-grade students who scored proficient 
or advanced on the language arts portion of CST during the baseline period and each of the two 
follow-up years." Figure 4.1 also shows the same data for those who scored below or far below 
basic. The white bars represent the outcomes of students in the BASRC focal districts, while the 
dark bars show the outcomes in the comparison districts. Figure 4.1 shows that in the BASRC 

1 1 As mentioned earlier, CST data allowed for just two years from which to determine a baseline average. 
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Figure 4.1 

Third-Grade Student Performance on the California Standards Test, Language Arts 
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SOURCE: MDRC calculations from district and school records from the California Department of 
Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 
2000-2001 and 2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003- 
2004) is the second follow-up school year. The estimates in the table represent averages across all 
districts, regardless of size. 
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focal districts, the percentage of students performing at or above the proficient level declined 
over time. In the years immediately preceding implementation of the focal reform strategy, an 
average of 43 percent of third-graders in the BASRC focal districts performed at or above profi- 
cient on the language arts portion of the CST. In follow-up Years 1 and 2, district proficiency 
levels were 41 percent and 37 percent, respectively. 12 In the comparison districts, the baseline 
average proficiency rate was 41 percent, the average in Year 1 was 43 percent, and the average 
in Year 2 was 39 percent. In short, over the first two years of the focal strategy, proficiency rates 
in the BASRC focal districts declined by approximately 6 percentage points, compared with a 
decline of 2 percentage points in the comparison districts. It is possible that the difference in the 
deviations from baseline suggests a small negative association between student achievement 
and the BASRC focal reforms. However, the difference is not statistically significant and may 
be due to chance. 13 

Figure 4.1 also illustrates a slight increase in the percentage of third-grade students per- 
forming below basic in the BASRC focal districts. In particular, 27 percent of students at BASRC 
focal districts scored below or far below basic during the baseline period, but this percentage in- 
creased to 30 percent by spring 2004. At the same time, the percentage of students scoring below 
or far below basic in the comparison districts essentially stayed the same. Again, the difference 
between focal districts and the comparison districts’ deviation from baseline is small and not sta- 
tistically significant. In other words, the association between BASRC focal reforms and progress 
over time is negative, but the difference is small and could be due to chance. 

One explanation for this decline in test scores is that students in all of the districts were 
still adjusting to recent changes in the CSTs. The pattern is also found across all districts in the 
Bay Area and across all districts in the state (as shown in Appendix C). On the other hand, the 
patterns do not suggest that the focal strategy was an effective means of improving third-grade 
achievement. 

Figure 4.2 presents the same CST outcomes for cohorts of fifth-grade students. Interest- 
ingly, the figures suggest improvements in language arts achievement in the focal districts at 
both points on the achievement scale. The average percentage of focal district students scoring 
at or above proficient increased from 38 percent to 5 1 percent by the end of follow-up Year 2. A 
similar change, from 39 percent to 50 percent, occurred in the comparison districts. These pat- 
terns are in line with upward trends across the Bay Area (and across the state as well). Figure 
4.2 also suggests that the BASRC focal districts reduced the number of students performing 

12 The number of questions on the CST changed somewhat between the last year of baseline and the first 
year of follow-up; therefore, small changes during this time should be interpreted with caution. The test did not 
change between follow-up Year 1 and follow-up Year 2. 

l3 By statistically significant, the authors mean that there is less than a 10 percent chance that the actual dif- 
ference between progress at the program and comparison groups is actually zero. 
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Figure 4.2 

Fifth-Grade Student Performance on the California Standards Test, Language Arts 
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SOURCE: MDRC calculations from district and school records from the California Department of 
Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 
2000-2001 and 2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003- 
2004) is the second follow-up school year. The estimates in the table represent averages across all 
districts, regardless of size. 
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below basic, and that, in Year 2, these changes may have outpaced those that occurred in the 
comparison districts. In particular, the percentage of students scoring below basic dropped by 4 
percentage points in the focal districts compared with essentially no change in the comparison 
districts, suggesting a net difference of 4 percentage points in the amount of progress in the fo- 
cal districts and their comparisons. 

Together these two graphs suggest a positive trend in the percentage of fifth-grade stu- 
dents meeting state standards in language arts. They also suggest the possibility of a positive 
association between the BASRC focal strategy and language arts achievement among fifth- 
graders, particularly among lower-performing students. However, the differences between pro- 
gress in the focal and comparison districts are relatively small and not statistically significant. 



Student Performance on the CAT-6 

The next set of figures shows achievement levels on California’s norm-referenced tests. 
As discussed above, the state of California administered the SAT-9 in the several years prior to 
BASRC implementation, and changed to the CAT-6 in the first year of the BASRC focal re- 
forms (spring 2003). As students and schools get used to new assessments, it is logical to expect 
percentile scores to drop when a new test is introduced, and improve in the years following the 
first administration of a new test. 14 Moreover, the CAT-6 and the SAT-9 are developed and pub- 
lished by two different companies and use different national groups of students as the compari- 
son groups. 15 As a result, it is difficult to interpret changes in test scores over time. However, 
absent any program effect, this change in test instruments should affect BASRC focal districts 
and comparison districts the same way. Therefore, to the extent that the focal strategy had a sys- 
tematic effect on student performance, we might still observe differences in the extent of these 
changes in test scores over time. 

Figure 4.3 presents the percentage of third-grade students who performed at or above the 
50th percentile and below the 25th percentile on the nationally norm-referenced tests administered 
in California. During the preimplementation period, the outcomes for the focal districts and the 
comparison districts looked nearly identical. In particular, approximately 60 percent of students in 
the focal districts and the comparison schools scored above the 50th percentile on the SAT-9. In 
the first two years of the BASRC focal reforms, when the CAT-6 administration began, the per- 
centage of students performing above the 50th percentile in both the focal and non-focal districts 
dropped to approximately 43 percent. This most likely does not reflect a substantive change in 



14 Koretz (2002) and Linn (2000). 

'’California Department of Education, Standardized Testing and Reporting (STAR) Results Web site 
(2005). 
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Figure 4.3 

Third-Grade Student Performance on Nationally Norm-Referenced 
Tests (SAT-9/CAT-6), Reading 



Percentage Scoring at or Above 50th Percentile 




Baseline Year 1 Year 2 Baseline Year 1 Year 2 



Focal Districts Comparison Districts 



Percentage Scoring Below 25th Percentile 




Baseline Year 1 Year 2 Baseline Year 1 Year 2 



Focal Districts Comparison Districts 

SOURCE: MDRC calculations from district and school records from the California Department of Education. 
NOTES: The Stanford Achievement Test (SAT-9) was replaced by the California Achievement Test (CAT-6) in 
the 2002-2003 school year. The baseline years for the SAT-9, language arts, consist of school years 1999-2000, 
2000-2001, and 2001-2002. Year 1 (2002-3003) is the first follow-up school year and Year 2 (2003-2004) is the 
second follow-up school year for the CAT-6. The estimates in the table represent averages across all districts, 
regardless of size. 
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teaching and learning, but a drop in scores due to the change in tests. However, there were no dif- 
ferences between the patterns in the BASRC focal districts versus their comparison counterparts. 
Figure 4.3 shows the same pattern for students who scored below the 25th percentile. 

Figure 4.4 presents the same results for fifth-grade students. It shows that in the first fol- 
low-up year, the percentage of students in focal and non-focal districts who exceeded the 50th 
percentile fell by the same amount (about 9 percentage points) in the focal districts as well as in 
the comparisons. But in the second year of the focal strategy, scores in the focal districts in- 
creased, while those in the comparison districts remained the same. In particular, the percentage 
exceeding the national norm in the focal districts increased to 50 percent, 6 percentage points 
lower than the baseline average. On the other hand, the average remained at approximately 48 
percent in the comparison districts, 9 points lower than their baseline average. Figure 4.4 also 
shows a similar story for students who scored below the 25th percentile, although the differ- 
ences are smaller. In both of the cases illustrated in Figure 4.4, however, the differences in de- 
viations from baseline are not statistically significant. 



Findings for Key Student Subgroups 

A key part of BASRC’s mission is “to raise student achievement and narrow the gap in 
performance between children of color, poor children, English Language Learners and their 
higher-achieving peers; and to create a more equitable system of schools.” 16 This section ex- 
plores whether the focal strategy is associated with changes in student achievement for students 
for which BASRC aims to accelerate achievement. 

This section presents the same figures for performance on the CST as for the full sam- 
ples of third- and fifth-grade students, but for each of three subgroups: economically disadvan- 
taged students (defined as students who qualify to receive a free or reduced-price lunch pro- 
gram), black and Hispanic students, and English Language Learners. 

Economically Disadvantaged Students 

• While the performance of economically disadvantaged third-graders in the 
focal districts declined, the performance among economically disadvantaged 
fifth-grade students increased in the years after implementation of the 
BASRC focal reforms. 17 



16 BASRC Web site (2003). 

17 As illustrated in Appendix C, this pattern is observed across the Bay Area as a whole as well as through- 
out the state of California. 
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Figure 4.4 

Fifth-Grade Student Performance on Nationally Norm-Referenced 
Tests (SAT-9/CAT-6), Reading 

Percentage Scoring at or Above 50th Percentile 
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SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The Stanford Achievement Test (SAT-9) was replaced by the California Achievement Test (CAT-6) in 
the 2002-2003 school year. The baseline years for the SAT-9, language arts, consist of school years 1999-2000, 
2000-2001 and 2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003-2004) is the 
second follow-up school year for the CAT-6. The estimates in the table represent averages across all districts, 
regardless of size. 
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• The improvements among lower-performing fifth-grade students were larger 
in the focal districts than in the comparison districts, but only in Year 2. The 
differences were not statistically significant. 

Approximately 30 percent of students in the elementary schools in the BASRC focal dis- 
tricts receive a free or reduced-price lunch, a common proxy for economic disadvantage. This 
proportion is constant through all years in MDRC’s analysis, including both the baseline and fol- 
low-up periods. On average, economically disadvantaged students scored much lower than aver- 
age on state reading assessments. For example, during the baseline period, 38 percent of all fifth- 
graders in the focal districts scored proficient or above on the language arts section of the CST, but 
the same could be said of only 20 percent of economically disadvantaged fifth-graders. 

In the focal districts, the performance of economically disadvantaged third-grade stu- 
dents on the CST declined slightly over the follow-up period, while in the comparison districts, 
there was little change. This is shown in Figure 4.5. However, the pattern among fifth-graders is 
somewhat different. Figure 4.6 shows the percentage of economically disadvantaged fifth-grade 
students who scored proficient or above on the language arts portion of the CST, as well as the 
percentage of economically disadvantaged fifth-graders who scored below or far below basic. 
As the figure illustrates, test scores among these students improved during the first two years of 
focal reforms. In particular, the percentage of fifth-graders scoring at or above proficient im- 
proved from 20 percent to 31 percent by Year 2 in the BASRC focal districts, and from 19 per- 
cent to 30 percent by Year 2 in the comparison districts. At the same time, the percentage of 
economically disadvantaged students scoring below basic fell from approximately 36 percent to 
30 percent by Year 2 in the BASRC focal districts and from 36 percent to 34 percent by Year 2 
at the comparison schools. Interestingly, the differences in improvements favored the compari- 
son districts in Year 1. However, by the spring of Year 2, the differences in progress favored the 
BASRC focal districts. In both cases, the differences were not statistically significant. 

Black and Hispanic Students 

This section focuses on trends among black and Hispanic students, two groups for 
which there are often achievement gaps. 18 During the baseline period, more than 30 percent of 
students in the elementary schools in the BASRC focal districts were black or Hispanic. In gen- 
eral, these students performed slightly lower than average. 



18 See, for example, Phillips, Crouse, and Ralph (1998). 
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Figure 4.5 

Third-Grade Student Performance on the California Standards Test, Language Arts: 

Economically Disadvantaged Students 
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SOURCE: MDRC calculations from district and school records from the California Department of 
Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000- 
2001 and 2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003-2004) is the 
second follow-up school year. The estimates in the table represent averages across all districts, regardless of 
size. 
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Figure 4.6 

Fifth-Grade Student Performance on the California Standards Test, Language Arts: 

Economically Disadvantaged Students 
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SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 
and 2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003-2004) is the second 
follow-up school year. The estimates in the table represent averages across all districts, regardless of size. 
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• Among third-graders, performance among black and Hispanic students in 
both focal districts and the comparison group fell in Year 2 of BASRC im- 
plementation. Though the differences were not statistically significant, the 
declines in the percentage of black and Hispanic third-graders performing at 
the proficient or advanced level appeared to be somewhat larger in the 
BASRC focal districts. 

• Performance among black and Hispanic fifth-graders appeared to improve. 
Though the differences were not statistically significant, the reduction in the 
percentage of black and Hispanic fifth-graders performing at the lowest lev- 
els appeared to be somewhat larger in the BASRC focal districts. 

Figure 4.7 shows the percentage of black and Hispanic third-graders scoring at or above 
proficient on the CST, as well as the percentage scoring below basic. 19 The figure reflects over- 
all declines in achievement levels among black and Hispanic third-graders. The figure also 
shows that these declines are slightly larger in the focal districts. On the other hand, among 
fifth-graders, CST proficiency scores improved relative to the baseline. As shown in Figure 4.8, 
during the baseline period, 20 percent of black and Hispanic students in the focal districts scored 
as proficient or advanced. By the second year of implementation, proficiency rates among these 
black and Hispanic fifth-graders had improved by 10 percentage points. Proficiency rates 
among fifth-graders in the comparison districts followed a nearly identical trend, growing from 
21 percent in the baseline period to 32 percent in Year 2 of the reforms. 

Figure 4.8 also shows reductions in the percentage of black and Hispanic students per- 
forming at the lowest levels of achievement. As the figure illustrates, the percentage of fifth- 
grade students scoring below or far below basic declined from 36 percent to 29 percent over the 
first two years of the focal reforms, a 7 percentage point difference. The percentage of fifth- 
graders scoring below basic in the comparison districts fell from 36 percent to 32 percent over 
the same period. Again, while the differences between the focal districts’ improvement and 
comparison districts' improvement are small and not statistically significant, by the second year 
of implementation, they favor the BASRC focal reform districts. 



l9 Data for students scoring below or far below basic are not available for racial/ethnic categories in the 
2001-2002 school year. Therefore, the baseline average for this subgroup is based on only one year, 2000- 



2001 . 
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Figure 4.7 

Third-Grade Student Performance on the California Standards Test, Language Arts: 

Black and Hispanic Students 
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SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 
and 2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003-2004) is the second 
follow-up school year. The estimates in the table represent averages across all districts, regardless of size. 



53 



Bay Area School Reform Collaborative Focal Strategy Evaluation 

Figure 4.8 

Fifth-Grade Student Performance on the California Standards Test, Language Arts: 

Black and Hispanic Students 
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SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 
and 2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003-2004) is the second 
follow-up school year. The estimates in the table represent averages across all districts, regardless of size. 
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English Language Learners 

The final subgroup analysis looks at students classified as English Language Learners 
(ELL). In the BASRC focal districts, 23 percent of students are classified as English Language 
Learners in third grade and 16 percent in fifth grade. 

• Test scores among third-grade English Language Learners declined slightly 
over the course of the follow-up period. The declines were somewhat larger 
in the focal districts than in the comparison districts, but the differences in 
declines were not statistically significant. 

• Among English Language Learners in the fifth grade, student performance 
appeared to increase and outpace that of students in the comparison districts 
in both follow-up years, particularly at the lower end of the achievement 
spectrum. Though the magnitude of the differences could be considered pol- 
icy-relevant, the differences in increases between the two groups of districts 
are not statistically significant. 

Overall, English Language Learners performed at much lower levels than the full sam- 
ple of students in the focal districts. Lor example, during the baseline period, an average of 1 1 
percent of fifth-grade English Language Learners in the focal districts scored as proficient or 
advanced on the CST compared with 38 percent of all fifth-graders. 

Ligure 4.9 shows the percentage of ELL third-graders who scored proficient or above, 
as well as the percentage of ELL third-graders who scored below basic. Ligure 4. 10 shows the 
same data for fifth-grade ELL students. 

In general, the figures show that — among both third- and fifth-graders — the vast major- 
ity of ELL students did not score proficient, and there was not much change over the course of the 
two years since implementation of the BASRC focal reform strategy. However, the changes in the 
percentages of ELL students who performed below basic show interesting patterns. Among third- 
grade ELL students in the focal districts, the percentage of students scoring below basic averaged 
52 percent during the baseline period and remained at a similar level during the follow-up. In the 
comparison districts, the percent of ELL third-graders scoring below basic fell from 60 percent in 
the baseline period to 53 percent in the second year of the follow-up period. 

In the fifth grade, on the other hand, there were substantial changes in the percentage of 
ELL students performing at low levels of achievement in the focal districts. During the baseline 
period, 55 percent of ELL fifth-graders in the focal districts scored below or far below basic. By 
the second year of follow-up, only 41 percent of these students scored below or far below basic, 
a reduction of 14 percentage points. In the comparison districts, the percentage of students per- 
forming below basic fell by 2 percentage points by Year 2, from 52 percent to 50 percent. This 
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Figure 4.9 

Third-Grade Student Performance on the California Standards Test, Language Arts: 

English Language Learners 
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SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 
and 2001-2002. Year 1 (2002-2003)is the first follow-up school year and Year 2 (2003-2004) is the second follow 
up school year. The estimates in the table represent averages across all districts, regardless of size. 
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Figure 4.10 

Fifth-Grade Student Performance on the California Standards Test, Language Arts: 

English Language Learners 
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SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000- 
2001 and 2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003-2004) is the 
second follow-up school year. The estimates in the table represent averages across all districts, regardless of 
size. 
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suggests a positive association between participation in the BASRC focal reforms and im- 
provements among ELL fifth-graders. Though this 12 percentage point difference is large 
enough to be considered policy-relevant by many, it is not statistically significant. 



Conclusions 

As mentioned at the outset of the chapter, MDRC’s design is not sufficient to determine 
for certain whether a causal link exists between the BASRC focal strategy and changes in stu- 
dent achievement in participating districts. Nevertheless, the analysis sheds some light on the 
relationship between the intervention and progress in student performance on literacy tests. In 
the years following implementation of the BASRC focal reforms, achievement levels among 
third-grade students in the BASRC focal districts fell slightly relative to California state stan- 
dards. A similar pattern occurred among third-graders in similar districts throughout the Bay 
Area, as well as in the state as a whole. Though the differences were small and not statistically 
significant, the decline was somewhat greater in the BASRC focal districts than in similar dis- 
tricts throughout the Bay Area. This appeared to be the case among all third-grade students as 
well as among subgroups of third-grade students. 

The analysis of fifth-grade achievement results suggests a somewhat different pattern. 
Average CST scores among fifth-graders in the focal districts and in the comparison districts 
improved over time, mostly in terms of an increase in the proportion of students scoring profi- 
cient or advanced. The focal districts’ improvements in proficiency rates are actually smaller 
than the comparison districts’ in Year 1, but surpassed the comparison districts by Year 2. 
However, in both years, the focal districts’ reductions in the percentage of students scoring be- 
low basic marginally exceeded the comparison districts’ reduction. In general, however, the dif- 
ferences between improvements among fifth-grade performance in focal and comparison dis- 
tricts are small and not statistically significant. 

When focusing on fifth-grade achievement among economically disadvantaged students, 
black and Hispanic students, and English Language Learners by Year 2, performance in the focal 
districts appeared to surpass the improvements in the comparison districts. The differences were 
most evident with respect to reductions in the number of fifth-grade students performing at the 
lowest levels of achievement. However, these differences were generally small, generally limited 
to Year 2, and not statistically significant. As a result, MDRC cannot discern whether the ob- 
served patterns represent systematic differences between focal and non-focal districts or fluctua- 
tions driven by chance. Nevertheless, the fact that the differences in achievement patterns are con- 
sistent across all three disadvantaged subgroups suggests the possibility that the BASRC focal 
strategy is associated, albeit modestly, with improved achievement among low-performing fifth- 
graders. MDRC’s next report will include an additional year of follow-up, which will indicate 
whether the improvement patterns are sustained or even strengthened over time. 
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Chapter 5 

Conclusions and Implications 



The BASRC “theory of action” specifies a set of reforms built around coaching, net- 
working and collaboration, and the use of inquiry- or evidence-based decision-making in order 
to improve student achievement and reduce achievement gaps between different racial/ethnic 
groups and students with different socioeconomic backgrounds. The BASRC focal reform strat- 
egy intensifies efforts along these dimensions, while adding more coaching at both the district 
and the school levels. What the BASRC focal strategy does not do is prescribe the specific in- 
structional changes, or even specific instructional supports (for example, professional develop- 
ment focused on adopted curricula or coaching focused on particular instructional improve- 
ments) that should be put in place in schools or classrooms to bring about this progress. 

In other words, even in theory, the model' s proximity to what it intends to change — 
teaching and learning in the classroom — is limited. Other reforms, particularly school-based 
efforts such as Success for All, recommend a specific instructional strategy and a set of supports 
and training designed to supplement and reinforce the implementation of that strategy. More- 
over, some districts' reform agendas are focused — at least in part — on the adoption and im- 
plementation of, as well as professional development for, particular reading and math curricula . 1 
In contrast, the BASRC focal reform strategy does not focus on implementing particular class- 
room instructional strategies. Rather, the BASRC theory of action and the focal reform strategy 
emphasize a process through which schools and teachers can come to learn how best they can 
support improved teaching and learning for the students that they serve. 

The implementation data gathered by MDRC underscore this dynamic. The data suggest 
that the BASRC focal strategy primarily targeted district leadership, and that the intensity of the 
reforms tended to wane with their proximity to the classroom. While district-level coaching con- 
tinued throughout the implementation of the BASRC focal strategy, in most districts, school-level 
coaching by BASRC staff was among the first components of the strategy to fall by the wayside. 
Moreover, though BASRC was designed to support a process of inquiry, sharing, and collective 
problem-solving, teachers were still reluctant to allow observation or critique of their classroom 
practices as part of this process. Finally, though a goal of the BASRC focal strategy was to inten- 
sify the core BASRC reforms, it was unclear whether the reforms implemented resulted from the 
focal strategy. It is possible that the schools would have mounted many of these reforms even in 
the absence of the focal strategy (either as part of other/earlier phases of BASRC, or as a function 



Snipes, Doolittle, and Herlihy (2002). 
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of other school improvement efforts already underway). The implementation data indicate that 
many of the BASRC reform strategies were in fact implemented prior to the focal strategy at 
many of the schools in the BASRC focal districts. 

Given the nature of the reforms and the implementation patterns, perhaps it is not sur- 
prising, that, on average, no strong and pervasive association is found between districts’ partici- 
pation in the BASRC focal reform strategy and changes in average student proficiency rates on 
state-mandated literacy tests. At the third-grade level, student performance changed little and 
even declined throughout the sample over the first two years after implementation of the 
BASRC focal reforms. This finding is consistent with the hypothesis that both proximity to in- 
struction and intensity of reforms are necessary in order to generate improvement in student 
achievement. On the other hand, it can also be argued that, because systemic approaches such as 
the BASRC focal reform strategy require years to take root, they would have to be in place for a 
longer time to yield an effect. Moreover, advocates of this type of approach would argue that, 
once in place, such reforms have the potential for more powerful sustained effects on teaching 
and learning throughout school systems than do school-based reforms based on “one size fits 
all” instructional strategies. 

At the same time, though the results are not statistically significant, evidence presented 
in this report suggests the possibility that the BASRC focal strategy is associated with progress 
at the fifth-grade level, particularly among disadvantaged, minority, and lower-performing stu- 
dents. This is consistent with BASRC’ s goal of reducing achievement gaps among students of 
different racial and socioeconomic backgrounds. This finding is also consistent with bringing 
evidence-based decision-making to bear on regular assessment and attention toward “target” 
students, or those students who need additional support. In other words, it is possible that the 
dimensions of BASRC that relate most directly to student performance, particularly among low- 
performing students, were beginning to affect progress among the neediest students, or that 
other efforts in these reform-minded districts in addition to BASRC were making a difference 
for their neediest students. 

It is important to remember, however, that these differences were modest, generally 
limited to Year 2, and not statistically significant. Moreover, the evaluation design does not 
permit definitive causal inferences from these associations. It may be that the focal districts 
opted to participate in the focal strategy because they were focused on raising the achievement 
of lower-performing students. So, while the findings in this report may reflect the limitations of 
the study rather than those of the reform, there is insufficient evidence to conclude that the focal 
reforms actually made a difference, even for disadvantaged and minority students. 

It is possible that the BASRC focal strategy is associated with modest improvements for 
some students. Overall, however, the question remains as to why improvements in achievement 
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in the BASRC focal districts were not larger and more pervasive. The findings suggest two pos- 
sible interpretations. 

• The BASRC focal reforms were not intense enough to affect students’ aca- 
demic performance. 

While the core components of the BASRC focal reforms are potential drivers of im- 
proved student achievement, the implementation of the focal reform strategy may simply not 
have been intense enough to yield systematic changes in student performance. While the 
BASRC theory of action suggests that reforms should take place at every level of the system, 
the primary supports for the reform strategy are at the district level. It may be that the district- 
level reforms are not sufficiently focused on the core reforms or are not sufficiently concen- 
trated or consistent enough to lead to systematic changes throughout the district. As such, rather 
than generating reforms aimed at teaching and learning, the BASRC focal strategy may simply 
provide general support at the district level. To the extent that this is true, it suggests that, if 
BASRC is to have an impact, BASRC must increase the intensity of its district reform efforts, 
provide a consistent focus for district reforms, and perhaps provide additional reinforcement for 
the reforms at the school level. 

• The BASRC focal reform components are not effective levers for improving 
student achievement. 

On the other hand, it is possible that BASRC focal reforms were implemented with suf- 
ficient intensity, but that the reforms themselves simply are not effective strategies for improv- 
ing student achievement. The BASRC focal strategy did not make specific changes in teaching 
and learning the direct target of its intervention. Instead, it focused on a set of processes that 
were thought to lead to changes in teaching and learning. It can be argued that, in order to affect 
teaching and learning, the focal strategy must intervene in ways that are more proximal to the 
classroom. And while coaching and focal interactions may affect school and classroom practice, 
improvements in average proficiency rates on tests did not appear to be associated with the focal 
reforms. This suggests that coaching and focal interactions do not systematically improve teach- 
ing and learning over and above what would have happened without the program. In other 
words, to the extent that the BASRC focal reforms actually affect the classroom, the resulting 
changes may not be any more effective than support strategies already being undertaken in 
similar districts. Again, the analysis presented in MDRC’ s next report will explore this issue by 
examining the correlation between survey measures of school reform practices and changes in 
student performance since the implementation of BASRC. 

In sum, it can be argued that successful district reforms require intensity, a direct link to 
classroom instruction, and components that are effective drivers of improved teaching and 
learning. More research is needed to understand the extent to which the BASRC focal reforms 
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incorporated these characteristics. MDRC’s next report will shed light on these issues by report- 
ing on district-level field research and exploring both the intensity and the focus of district-level 
reforms in the BASRC focal districts. It will also examine the relationship between specific 
BASRC reform practices and school-level changes in student performance. Finally, the next 
report will follow progress in the BASRC focal and comparison districts for an additional 
school year. This will allow an assessment of whether the improvements in student outcomes in 
the focal districts grow, fade, or are sustained over time, and whether these changes continue to 
outpace those in the comparison districts. 
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Appendix A 

Background Information on the California Standards 
Tests, the Stanford Achievement Test, and 
the California Achievement Tests 




The California Standards Tests 



The California Standards Tests (CSTs) are a series of tests administered to California 
public school students in grades 2-1 1. 1 While most questions are multiple-choice, students in the 
fourth and seventh grades complete a composition or writing section. CSTs include somewhere 
between 65 and 75 language arts questions depending upon the grade and 65 mathematics ques- 
tions. Strands for the language arts questions include: word analysis, reading comprehension, lit- 
erary response and analysis, writing strategies, and written conventions. In addition, students in 
grades 8, 10, and 1 1 answer 60 to 80 history/social science questions, and those in grades 5 and 9- 
1 1 answer approximately 60 science questions. Tests are typically administered over the course of 
one to three days. 

CSTs are standards-based or criterion-referenced tests. In other words, the tests attempt 
to measure whether students are meeting the content standards adopted by the California De- 
partment of Education. Results are reported according to a five-point proficiency scale as ad- 
vanced, proficient, basic, below basic, and far below basic (with the exception of the year 2000, 
in which CST scores were reported by the average number correct and total number possible). 



Stanford Achievement Test and California Achievement Tests 

Both the Stanford Achievement Test, Ninth Edition (SAT-9), administered from 1998 
to 2002 in California public schools, and the California Achievement Tests, Sixth Edition 
(CAT-6), administered from 2003 to the present, are nationally norm-referenced tests. In other 
words, results for the SAT-9 and CAT-6 report how well students compare with a nationwide 
sample of students. Scores are reported in terms of a raw score, which is converted into a scaled 
score based on test difficulty, and a national percentile rank. 

The SAT-9 and CAT-6 are in many ways similar in content and format. Both tests consist 
entirely of multiple-choice questions that are unchanged from year to year, with tests for grades 2- 
1 1 including reading, language, and mathematics content sections. Tests for grades 2-8 also in- 
clude a spelling section. Both tests for grades 9-11 include a science section, wh il e the SAT-9 for 
grades 9-11 also includes a social sciences section. While there are indeed some similarities, re- 
sults for the SAT-9 and CAT-6 are not directly comparable to one another since the different pub- 
lishers (Harcourt and McGraw-Hill, respectively) produced each test at different times, employing 
different national samples, and created tests of different difficulty levels. 



'Sources referenced for this appendix include the California Department of Education Web pages on test- 
ing ( http://www.cde.ca.gov/ta/tg/ ) and the California Standardized Testing and Reporting Web pages from 
2000, 2002, and 2004 ( http://star.cde.ca.gov/ ). 
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Appendix B 

Selection of Comparison Districts 




Comparison districts were selected with the goal of finding districts that were as similar 
as possible to the BASRC focal districts in terms of student demographics and the history of 
academic performance. As the analysis focused on elementary achievement through the fifth 
grade, MDRC selected comparison districts according to average demographic characteristics 
and achievement levels among the fifth-grade students in each district. Based on annual data 
obtained from the California Department of Education, demographic characteristics and 
achievement levels were averaged across three baseline school years (1999-2000, 2000-2001, 
and 2001-2002), and districts were matched based on those averages. 1 The following are the 
criteria used to identify comparison districts for each of the five focal districts: 

• The district existed and had more than 10 students in fifth grade in all base- 
line years. 

• The district is in the San Francisco Bay Area (in one of the following counties: 
Alameda, Contra Costa, Marin, San Francisco, San Mateo, and Santa Clara). 

• The baseline mean percentage of fifth-grade students scoring proficient or 
above on the language arts portion of the California Standards Tests (CST) is 
within 20 percent of the baseline mean percentage at the focal district. 2 

• The baseline mean percentage of fifth-grade students scoring at or above the 
50th percentile on the reading portion of the SAT-9 is within 20 percent of 
the baseline mean percentage at the focal district. 3 

• Fooking at the most prevalent racial/ethnic group among fifth-grade students 
in the focal district, the average percentage of that racial/ethnic group in a 
comparison district is within 20 percentage points of the focal district average. 

• Fooking at the second most prevalent racial/ethnic group among fifth-grade 
students in the focal district, the average percentage of that racial/ethnic 
group in a comparison district is within 20 percentage points of the focal dis- 
trict average. 

• The number of fifth-grade students is within 50 percent of the number of 
fifth-grade students enrolled in the focal district. 



'California Standards Test data are available only in 2000-2001 and 2001-2002 for the language arts sec- 
tion and 2001-2002 for the math section. Therefore, district selection was based only on the average across 
these years. 

2 Note that “language arts” is the broadest subtest on the CST and includes reading, spelling, etc. 

3 Note that “reading” is the broadest strand on the SAT-9 and includes language arts and spelling. 
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As discussed in Chapter 4, the process for selecting comparison districts produced the 
following results. Overall, 15 distinct districts wi t hin the Bay Area matched with the five focal 
districts, with some comparison districts matching with more than one focal district. Each of the 
five BASRC focal districts matched with between two and five comparison districts. Overall, 
the two sets of districts look very similar. 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 

Appendix Table B.l 

Characteristics of BASRC Focal and Comparison Districts by Year, Third-Grade Students 



Characteristic 


2000 




2001 






2002 


All 


years 


Focal Comparison 


Focal Comparison 


Focal 


Comparison 


Focal 


Comparison 


Demographic characteristics of 3rd-grade students 


















Average number of 3rd-grade students 


512 


610 


511 


599 


492 


592 


505 


601 


Average number of elementary schools 3 


8 


6 


8 


6 


8 


7 


8 


6 


Race/Ethnicity (%) 


















White 


42.0 


49.9 


38.8 


47.2 


38.9 


45.8 


39.9 


47.7 


Hispanic 


27.2 


27.5 


29.4 


29.2 


30.0 


29.8 


28.9 


28.8 


Asian 


21.0 


16.6 


21.2 


17.6 


23.2 


17.9 


21.8 


17.4 


Black 


5.7 


4.9 


5.7 


4.4 


5.1 


5.1 


5.5 


4.8 


Other 


4.0 


1.0 


4.9 


1.6 


2.9 


1.4 


3.9 


1.3 


English language learners (%) 


19.8 


16.9 


23.4 


17.0 


24.8 


20.7 


22.7 


18.2 


Free/reduced-price lunch (%) 


29.3 


25.6 


28.9 


24.1 


31.1 


24.4 


29.8 


24.7 


Achievement levels of 3rd-grade students 


















California Standards Test 


















Language arts 


















Proficient or advanced (%) 






41.7 


38.6 


44.4 


43.8 


43.1 


41.2 


Basic (%) 






30.6 


32.2 


29.2 


29.6 


29.9 


30.9 


Below or far below basic (%) 






27.7 


29.2 


26.2 


26.7 


26.9 


28.0 


Mean scale score 










341.8 


339.3 


341.8 


339.3 


Math 


















Proficient or advanced (%) 










49.4 


46.2 


49.4 


46.2 


Basic (%) 










27.2 


26.8 


27.2 


26.8 


Below or far below basic (%) 










23.6 


26.8 


23.6 


26.8 


Mean scale score 










350.8 


346.1 


350.8 


346.1 
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Appendix Table B.l (continued) 







2000 




2001 




2002 


All 


years 


Characteristic 


Focal 


Comparison 


Focal 


Comparison 


Focal 


Comparison 


Focal 


Comparison 


Achievement levels of 3rd-erade students 

Stanford Achievement Test, Ninth Edition 
Reading 


At or above 50th national percentile ranking (%) 


59.4 


57.8 


59.0 


59.5 


61.2 


61.1 


59.9 


59.5 


25th national percentile ranking (%) 


20.8 


23.1 


23.8 


22.2 


20.8 


21.9 


21.8 


22.4 


Below 25th national percentile ranking (%) 


19.8 


19.1 


17.2 


18.4 


18.0 


17.0 


18.3 


18.2 


Mean scale score 


623.6 


622.8 


627.8 


624.5 


628.0 


626.7 


626.4 


624.7 


Math 


At or above 50th national percentile ranking (%) 


63.8 


66.0 


70.0 


68.6 


73.2 


69.7 


69.0 


68.1 


25th national percentile ranking (%) 


21.0 


19.1 


18.0 


17.8 


15.6 


17.7 


18.2 


18.2 


Below 25th national percentile ranking (%) 


15.2 


14.9 


12.0 


13.6 


11.2 


12.6 


12.8 


13.7 


Mean scale score 


615.5 


615.9 


622.2 


619.1 


624.7 


620.9 


620.8 


618.6 


Total number of districts 


5 


14 


5 


14 


5 


14 


5 


14 



SOURCE: MDRC calculations from district and school records from the California Department of Education. 



NOTES: Comparison district columns represent the average of the average across each comparison district cluster. Note that some comparison districts 
match with more than one focal district. The estimates in the table represent averages across all districts, regardless of size. 

‘‘Elementary schools include any school with a fifth-grade class, since comparison districts were selected from fifth-grade outcomes. 
b These National Percentile Ranks correspond to the percentage of students in the district with scores corresponding to those of students in the top 50%, 
25%, and below 25% of the national sample. 




Bay Area School Reform Collaborative Focal Strategy Evaluation 
Appendix Table B.2 

Characteristics of BASRC Focal and Comparison Districts by Year, Fifth-Grade Students 



Characteristic 


2000 




2001 




2002 


All 


years 


Focal Comparison 


Focal Comparison 


Focal 


Comparison 


Focal 


Comparison 


Demographic characteristics of 5th-grade students 


















Average number of 5th-grade students 


490 


575 


515 


617 


492 


604 


499 


599 


Average number of elementary schools' 1 


8 


6 


8 


6 


8 


7 


8 


6 


Race/Ethnicity (%) 


















White 


44.5 


50.7 


43.8 


48.4 


41.1 


48.5 


43.1 


49.2 


Hispanic 


25.8 


25.3 


25.4 


27.7 


27.4 


27.5 


26.2 


26.8 


Asian 


21.6 


17.3 


21.4 


16.9 


22.7 


18.0 


21.9 


17.4 


Black 


6.2 


5.5 


6.8 


5.4 


5.7 


4.9 


6.2 


5.3 


Other 


1.9 


1.3 


2.6 


1.7 


3.1 


1.0 


2.5 


1.3 


English language learners (%) 


15.2 


14.3 


14.9 


12.8 


16.4 


14.8 


15.5 


14.0 


Free/reduced-price lunch (%) 


29.3 


25.6 


28.9 


24.1 


31.1 


24.4 


29.8 


24.7 


Achievement levels of 5th-grade students 


















California Standards Test 


















Language arts 


















Proficient or advanced (%) 






36.7 


37.1 


38.8 


40.2 


37.8 


38.6 


Basic (%) 






39.1 


40.2 


40.4 


41.9 


39.8 


41.1 


Below or far below basic (%) 






24.1 


22.7 


21.0 


17.8 


22.6 


20.3 


Mean scale score 










338.4 


341.1 


338.4 


341.1 


Math 


















Proficient or advanced (%) 










37.2 


36.9 


37.2 


36.9 


Basic (%) 










32.6 


33.5 


32.6 


33.5 


Below or far below basic (%) 










30.4 


29.6 


30.4 


29.6 


Mean scale score 










335.6 


335.7 


335.6 


335.7 
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Appendix Table B.2 (continued) 







2000 




2001 




2002 


All 


years 


Characteristic 


Focal 


Comparison 


Focal 


Comparison 


Focal 


Comparison 


Focal 


Comparison 


Achievement levels of 5th-erade students 

Stanford Achievement Test, Ninth Edition 
Reading 


At or above 50th national percentile ranking (%) 


55.4 


57.1 


56.2 


57.6 


56.4 


59.1 


56.0 


57.9 


25th national percentile ranking (%) 


24.6 


23.8 


23.2 


22.1 


22.4 


23.4 


23.4 


23.1 


Below 25th national percentile ranking (%) 


20.0 


19.1 


20.6 


20.3 


21.2 


17.5 


20.6 


19.0 


Mean scale score 


658.7 


659.4 


659.1 


658.7 


659.6 


661.0 


659.1 


659.7 


Math 


At or above 50th national percentile ranking (%) 


61.2 


59.4 


63.2 


64.0 


66.2 


67.2 


63.5 


63.5 


25th national percentile ranking (%) 


20.0 


20.0 


16.8 


17.3 


16.4 


16.4 


17.7 


17.9 


Below 25th national percentile ranking (%) 


18.8 


20.6 


20.0 


18.7 


17.4 


16.4 


18.7 


18.6 


Mean scale score 


656.1 


655.3 


659.5 


659.6 


663.1 


663.4 


659.6 


659.4 


Total number of districts 


5 


14 


5 


14 


5 


14 


5 


14 



SOURCE: MDRC calculations from district and school records from the California Department of Education. 



NOTES: Comparison district columns represent the average of the average across each comparison district cluster. Note that some comparison districts match 
with more than one focal district. The estimates in the table represent averages across all districts, regardless of size. 

‘‘Elementary schools include any school with a fifth-grade class, since comparison districts were selected from fifth-grade outcomes. 
b These National Percentile Ranks correspond to the percentage of students in the district with scores corresponding to those of students in the top 50%, 
25%, and below 25% of the national sample. 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 
Appendix Figure C.l 

Third- Grade Student Performance on the California Standards Test, Language Arts: 
Focal, Bay Area, and California Districts 



Percentage Scoring Proficient or Advanced □ BASRC Focal Districts 




Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 




Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 



SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 and 
2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003-2004) is the second follow-up 
school year. The estimates in the table represent averages across all districts, regardless of size. 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 
Appendix Figure C.2 

Fifth-Grade Student Performance on the California Standards Test, Language Arts: 
Focal, Bay Area, and California Districts 
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Percentage Scoring Proficient or Advanced 



□ BASRC Focal Districts 
■ Comparison Districts 
0 All Bay Area Districts 
H All California Districts 




Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 




SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 
and 2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003-2004) is the second follow- 
up (2003-2004) school year. The estimates in the table represent averages across all districts, regardless of size. 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 
Appendix Figure C.3 

Third-Grade Student Performance on Norm-Referenced Tests (SAT-9/CAT-6), Reading: 

Focal, Bay Area, and California Districts 
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Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 




SOURCE: MDRC calculations from district and school records from the California Department of Education. 
NOTES: The Stanford Achievement Test (SAT-9) was replaced by the California Achievement Test (CAT-6) in 
the 2002-2003 school year. The baseline years for the SAT-9, language arts, consist of school years 1999-2000, 
2000-2001 and 2001-2002. Year 1 (2002-2003) is the first follow-upschool year and Year 2 (2003-2004) is the 
second follow-up school year for the CAT-6. The estimates in the table represent averages across all districts, 
regardless of size. 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 
Appendix Figure C.4 

Fifth-Grade Student Performance on Norm-Referenced Tests (SAT-9/CAT-6), Reading: 

Focal, Bay Area, and California Districts 




Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 




Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 Baseline Year 1 Year 2 



SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The Stanford Achievement Test (SAT-9) was replaced by the California Achievement Test (CAT-6) in 
the 2002-2003 school year. The baseline years for the SAT-9, language arts, consist of school years 1999-2000, 
2000-2001 and 2001-2002. Year 1 (2002-2003) is the first follow-up school year and Year 2 (2003-2004) is the 
second follow-up school year for the CAT-6. The estimates in the table represent averages across all districts, 
regardless of size. 
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Appendix D 

Analysis of Achievement Outcomes 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 

Appendix Table D.l 

Third-Grade Outcomes, California Standards Test, Language Arts Analysis 



Outcome 




I. Outcome Levels Compared to Baseline Year and Follow-up Years 






BASRC schools 




Comparison schools 




Baseline 


Year 1 


Year 2 


Baseline 


Year 1 


Year 2 


Mean scaled score 


341.8 


338.0 


334.5 


339.3 


337.5 


334.8 


Deviation from baseline average 




-3.9 


-7.3 




-1.8 


-4.5 


Below basic 


26.9 


27.8 


29.8 


28.0 


27.0 


28.6 


Deviation from baseline average 




0.9 


2.9 




-1.0 


0.7 


Proficient or advanced 


43.1 


41.0 


37.2 


41.2 


43.1 


38.8 


Deviation from baseline average 




-2.1 


-5.9 




1.9 


-2.4 



Outcome 


II. Difference in Deviation from 
the Baseline 


III. Percent Differences a 


Year 1 


Year 2 


Year 1 


Year 2 


Mean scaled score 


-2.1 


-2.7 


-1% 


-1% 


p-value 


0.8 


0.7 






Below basic 


1.8 


2.2 


7% 


8% 


p-value 


0.6 


0.6 






Proficient or advanced 


-3.9 


-3.4 


-9% 


-8% 


p-value 


0.3 


0.4 







SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 and 2001-2002. Year 1 (2002-2003) 
is the first follow-up school year and Year 2 (2003-2004) is the second follow-up school year. The estimates in the table represent averages across all 
districts, regardless of size. 

“Percent differences are calculated from the baseline level. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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Appendix Table D.2 

Fifth-Grade Outcomes, California Standards Test, Language Arts Analysis 



Outcome 




I. Outcome Levels Compared to Baseline Year and Follow-up Years 






BASRC schools 




Comparison schools 




Baseline 


Year 1 


Year 2 


Baseline 


Year 1 


Year 2 


Mean scaled score 


338.4 


341.0 


352.4 


341.1 


343.3 


349.0 


Deviation from baseline average 




2.6 


14.1 




2.2 


7.9 


Below basic 


22.6 


21.6 


18.6 


20.3 


20.4 


20.6 


Deviation from baseline average 




-1.0 


-4.0 




0.2 


0.3 


Proficient or advanced 


37.8 


42.8 


51.2 


38.6 


44.4 


50.0 


Deviation from baseline average 




5.0 


13.4 




5.8 


11.4 



Outcome 


II. Difference in Deviation from 
the Baseline 


III. Percent Difference 11 


Year 1 


Year 2 


Year 1 


Year 2 


Mean scaled score 


0.4 


6.2 


0% 


2% 


p-value 


0.9 


0.3 






Below basic 


-1.1 


-4.3 


-5% 


-19% 


p-value 


0.7 


0.2 






Proficient or advanced 


-0.8 


2.0 


-2% 


5% 


p-value 


0.9 


0.6 







SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 and 2001-2002. Year 1 (2002-2003) is 
the first follow-up school year and Year 2 (2003-2004) is the second follow-up school year. The estimates in the table represent averages across all 
districts, regardless of size. 

‘‘Percent differences are calculated from the baseline level. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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Appendix Table D.3 

Third-Grade Outcomes, English Language Learners, California Standards Test, Language Arts Analysis 



Outcome 




I. Outcome Levels Compared to Baseline Year and Follow-up Years 






BASRC schools 




Comparison schools 




Baseline 


Year 1 


Year 2 


Baseline 


Year 1 


Year 2 


Mean scaled score 


303.2 


299.8 


300.4 


298.0 


301.0 


299.4 


Deviation from baseline average 




-3.4 


-2.8 




3.0 


1.4 


Below basic 


52.0 


54.6 


52.0 


59.7 


51.0 


52.7 


Deviation from baseline average 




2.6 


0.0 




-8.6 


-7.0 


Proficient or Advanced 


15.9 


14.2 


13.2 


11.6 


16.7 


14.2 


Deviation from baseline average 




-1.7 


-2.7 




5.1 


2.6 



Outcome 


II. Difference in Deviation from 
the Baseline 


III. Percent Differences a 


Year 1 


Year 2 


Year 1 


Year 2 


Mean scaled score 


-6.4 


-4.2 


-2% 


-1% 


p-value 


0.5 


0.7 






Below basic 


11.2 


7.0 


22% 


13% 


p-value 


0.2 


0.4 






Proficient or Advanced 


-6.8 


-5.3 


-43% 


-33% 


p-value 


0.2 


0.3 







SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 and 2001-2002. Year 1 (2002-2003) 
is the first follow-up school year and Year 2 (2003-2004) is the second follow-up school year. The estimates in the table represent averages across 
all districts, regardless of size. 

J Percent differences are calculated from the baseline level. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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Bay Area School Reform Collaborative Focal Strategy Evaluation 

Appendix Table D.4 



Fifth-Grade Outcomes, English Language Learners, California Standards Test, Language Arts Analysis 



Outcome 




I. Outcome Levels Compared to Baseline Year and Follow-up Years 






BASRC schools 




Comparison schools 




Baseline 


Year 1 


Year 2 


Baseline 


Year 1 


Year 2 


Mean scaled score 


301.5 


307.3 


311.9 


299.4 


308.4 


304.2 


Deviation from baseline average 




5.8 


10.4 




9.0 


4.8 


Below basic 


54.6 


43.4 


40.8 


51.5 


48.4 


49.4 


Deviation from baseline average 




-11.2 


-13.8 




-3.1 


-2.1 


Proficient or Advanced 


10.9 


14.0 


16.6 


7.3 


13.7 


13.8 


Deviation from baseline average 




3.1 


5.7 




6.4 


6.5 



Outcome 


II. Difference in Deviation from 
the Baseline 


III. Percent Differences a 


Year 1 


Year 2 


Year 1 


Year 2 


Mean scaled score 


-3.2 


5.6 


-1% 


2% 


p-value 


0.7 


0.5 






Below basic 


-8.1 


-11.7 


-15% 


-21% 


p-value 


0.4 


0.2 






Proficient or Advanced 


-3.3 


-0.8 


-30% 


-7% 


p-value 


0.5 


0.9 







SOURCE: MDRC calculations from district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 and 2001-2002. Year 1 (2002-2003) is 
the first follow-up school year and Year 2 (2003-2004) is the second follow-up school year. The estimates in the table represent averages across all 
districts, regardless of size. 

J Percent differences are calculated from the baseline level. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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Appendix Table D.5 

Third-Grade Outcomes, Economically Disadvantaged Students, California Standards Test, Language Arts Analysis 



Outcome 




I. Outcome Levels Compared to Baseline Year and Follow-up Years 






BASRC schools 




Comparison schools 




Baseline 


Year 1 


Year 2 


Baseline 


Year 1 


Year 2 


Mean scaled score 


314.3 


310.9 


308.6 


309.9 


307.8 


306.1 


Deviation from baseline average 




-3.4 


-5.7 




-2.1 


-3.8 


Below basic 


42.4 


44.8 


44.6 


46.7 


45.7 


46.5 


Deviation from baseline average 




2.4 


2.2 




-1.0 


-0.2 


Proficient or Advanced 


22.5 


21.2 


18.8 


19.5 


22.9 


19.4 


Deviation from baseline average 




-1.3 


-3.7 




3.4 


-0.1 



Outcome 


II. Difference in Deviation from 
the Baseline 


III. Percent Differences “ 


Year 1 


Year 2 


Year 1 


Year 2 


Mean scaled score 


-1.3 


-1.9 


0% 


-1% 


p-value 


0.9 


0.8 






Below basic 


3.4 


2.4 


8% 


6% 


p-value 


0.5 


0.7 






Proficient or Advanced 


-4.7 


-3.6 


-21% 


-16% 


p-value 


0.2 


0.4 







SOURCE: MDRC calculations front district and school records from the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 and 2001-2002. Year 1 (2002-2003) 
is the first follow-up school year and Year 2 (2003-2004) is the second follow-up school year. The estimates in the table represent averages across all 
districts, regardless of size. 

J Percent differences are calculated from the baseline level. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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Bay Area Reform School Collaborative Focal Strategy Evaluation 

Appendix Table D.6 

Fifth-Grade Outcomes, Economically Disadvantaged Students, California Standards Test, Language Arts Analysis 



Outcome 




I. Outcome Levels Compared to Baseline Year and Follow-up Years 






BASRC schools 




Comparison schools 




Baseline 


Year 1 


Year 2 


Baseline 


Year 1 


Year 2 


Mean scaled score 


317.7 


319.5 


327.9 


318.7 


321.6 


323.4 


Deviation from baseline average 




1.8 


10.2 




2.9 


4.8 


Below basic 


35.7 


34.4 


29.6 


36.3 


34.2 


34.4 


Deviation from baseline average 




-1.3 


-6.1 




-2.2 


-1.9 


Proficient or advanced 


19.9 


23.6 


31.0 


18.6 


23.0 


29.7 


Deviation from baseline average 




3.7 


11.1 




4.3 


11.1 



Outcome 


II. Difference in Deviation from 
the Baseline 


III. Percent Differences 0 


Year 1 


Year 2 


Year 1 


Year 2 


Mean scaled score 


-1.1 


5.5 


0% 


2% 


p-value 


0.9 


0.4 






Below basic 


0.9 


-4.2 


2% 


-12% 


p-value 


0.9 


0.4 






Proficient or advanced 


-0.6 


0.0 


-3% 


0% 


p-value 


0.9 


1.0 







SOURCE: MDRC calculations from district and school records front the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 and 2001-2002. Year 1 (2002- 
2003) is the first follow-up school year and Year 2 (2003-2004) is the second follow-up school year. The estimates in the table represent averages 
across all districts, regardless of size. 

J Percent differences are calculated from the baseline level. 

Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 
1 percent; ** = 5 percent; * = 10 percent. 
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Appendix Table D.7 



Third-Grade Outcomes, Black and Hispanic Students, California Standards Test, Language Arts Analysis 



Outcome 




I. Outcome Levels Compared to Baseline Year and Follow-up Years 






BASRC schools 




Comparison schools 




Baseline 


Year 1 


Year 2 


Baseline 


Year 1 


Year 2 


Mean scaled score 


304.9 


310.8 


311.6 


311.3 


308.0 


307.4 


Deviation from baseline average 




5.9 


6.7 




-3.3 


-4.0 


Below basic 


39.0 


42.1 


42.9 


41.6 


41.1 


43.4 


Deviation from baseline average 




3.0 


3.9 




-0.5 


1.8 


Proficient or advanced 


23.7 


22.6 


19.2 


23.0 


25.1 


21.1 


Deviation from baseline average 




-1.2 


-4.5 




2.1 


-1.9 



Outcome 


II. Difference in Deviation from 
the Baseline 


III. Percent Differences “ 


Year 1 


Year 2 


Year 1 


Year 2 


Mean scaled score 


9.2 


10.6 


3% 


3% 


p-value 


0.4 


0.3 






Below basic 


3.6 


2.1 


9% 


5% 


p-value 


0.5 


0.7 






Proficient or advanced 


-3.3 


-2.6 


-14% 


-11% 


p-value 


0.4 


0.5 







SOURCE: MDRC calculations front district and school records from the California Department of Education. 

NOTES: The baseline years for the CST Language Arts consist of school years 2000-2001 and 2001-2002. Year 1 (2002-2003) is the first follow-up 
school year and Year 2 (2003-2004) is the second follow-up school year. The estimates in the table represent averages across all districts, 
regardless of size. 

J Percent differences are calculated from the baseline level. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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Appendix Table D.8 

Fifth-Grade Outcomes, Black and Hispanic Students, California Standards Test, Language Arts Analysis 



Outcome 




I. Outcome Levels Compared to Baseline Year and Follow-up Years 






BASRC schools 




Comparison schools 




Baseline 


Year 1 


Year 2 


Baseline 


Year 1 


Year 2 


Mean scaled score 


313.9 


320.3 


313.8 


315.5 


324.9 


325.4 


Deviation from baseline average 




6.4 


-0.1 




9.4 


9.9 


Below basic 


35.9 


34.8 


28.9 


36.0 


32.3 


31.5 


Deviation from baseline average 




-1.1 


-7.0 




-3.7 


-4.5 


Proficient or advanced 


20.1 


25.3 


29.8 


21.3 


27.4 


32.4 


Deviation from baseline average 




5.2 


9.7 




6.1 


11.1 



Outcome 


II. Difference in Deviation from 
the Baseline 


III. Percent Differences 0 


Year 1 


Year 2 


Year 1 


Year 2 


Mean scaled score 


-2.9 


-10.0 


-1% 


-3% 


p-value 


0.7 


0.2 






Below basic 


2.7 


-2.6 


1% 


-7% 


p-value 


0.6 


0.6 






Proficient or advanced 


-0.9 


-1.5 


-5% 


-7% 


p-value 


0.8 


0.7 







SOURCE: MDRC calculations from district and school records front the California Department of Education. 

NOTES: The baseline years for the California Standards Test, language arts, consist of school years 2000-2001 and 2001-2002. Year 1 (2002-2003) is 
the first follow-up school year and Year 2 (2003-2004) is the second follow-up school year. The estimates in the table represent averages across all 
districts, regardless of size. 

“Percent differences are calculated from the baseline level. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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About MDRC 



MDRC is a nonprofit, nonpartisan social policy research organization dedicated to learn- 
ing what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness 
of social and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best 
known for mounting rigorous, large-scale, real-world tests of new and existing policies 
and programs. Its projects are a mix of demonstrations (field tests of promising new pro- 
gram approaches) and evaluations of ongoing government and community initiatives. 
MDRC’s staff bring an unusual combination of research and organizational experience to 
their work, providing expertise on the latest in qualitative and quantitative methods and 
on program design, development, implementation, and management. MDRC seeks to 
learn not just whether a program is effective but also how and why the program’s effects 
occur. In addition, it tries to place each project’ s findings in the broader context of related 
research — in order to build knowledge about what works across the social and education 
policy fields. MDRC’s findings, lessons, and best practices are proactively shared with a 
broad audience in the policy and practitioner community as well as with the general pub- 
lic and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of pol- 
icy areas and target populations. Once known primarily for evaluations of state welfare- 
to-work programs, today MDRC is also studying public school reforms, employment 
programs for ex-offenders and people with disabilities, and programs to help low-income 
students succeed in college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Child Development 

• Improving Public Education 

• Promoting Successful Transitions to Adulthood 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the 
United Kingdom, MDRC conducts its projects in partnership with national, state, and 
local governments, public school systems, community organizations, and numerous pri- 
vate philanthropies. 




